What Xie Xian really cared about was not the 'sick ghost' mentioned by Xiao Second Lady, who abandoned Yang Shao for him in order to gain power--although the term 'sick ghost' was indeed harsh and hateful.
What he cared about was what Second Lady Xiao said about 'protecting the Xiao family'.
The Xiao family was at the height of their power. Even if they were involved in the battle for the crown prince, they would not be safe. Moreover, the Xiao family was on the same side. Even if they were defeated, the two families would end up the same.
The marriage between the Xiao and Xie families was clearly before the death of Noble Consort Liu. At least the Emperor did not reveal any intention of changing the heir.
He often accompanied the Emperor, so the Emperor would not hide this from him. He would definitely try everything possible to rope him in to carry the sedan chair for Xin 'an King. There was no way he could directly talk to Sikong Xiao.