This was why he wanted his mother to leave the house she shared with Charles and their marriage completely so that he could protect her. She was so wrapped up in being scared of his father, she never noticed what Eli had planned and achieved. He was not scared to go up the likes of his father. Eli wasn't scared to take on Charles in a one-on-one fight but he knew his father would never play fair.
However, Eli had studied his father for many years. If you want to break the man, you get rid of his wife. The slightest thought that Elaine would leave Charles made him practically lose his mind to the point where he wouldn't think things through. Charles was too obsessed with Elaine trying to make her love him.