After Geoffrey spoke those words, it took Charlton all his self-control not to react negatively. He knew that he was still powerless to lay any claim on Serena. So, he just laughed it off and responded with an 'of course.'
It was the first time that Geoffrey warned him directly and feelings of anger intermixed with guilt surfaced within him. Anger, because although Geoffrey was the crown prince, he was still not the king. Although his station was above him, he still needs his support in the future. Guilt, because he already did what he asked him not to do but again, he already made his choice.
He knew that Geoffrey will not let Serena go without a fight, but it was not him that he needs to convince. The engagement was political in nature, thus, so long as he can present his uncle, the king, a greater benefit in exchange for Serena's hand, he still has a chance for plan A to work.
