Rafael couldn't pinpoint when his hand moved to grab Joel's neck but he didn't mind that it was there. Joel lacked the ability to know when to shut up or what not to say to someone. He was almost curious about what was going on in Joel's head to think what he said was a good idea.
He knew what Joel meant by saying he was waiting for Annalise and her people to be turned into slaves. It was the same disgusting talk he had heard before of wanting the women for sexual acts. Another reason for him to strangle Joel was for mentioning slaves when it was not something the kingdom did anymore.