Saying that he didn't go directly towards 'The Mansion' but towards the couple café that Chu Fan was currently in.
After the matter that night, the girls of the café were still kept under surveillance at the Tianlong House. Only the manager, the most obeying toy of Chu Fan was left to return there, and start employing new girls.
Tianlong Yun was controlling her through Chu Fan. He didn't want to hurt the other girls if not necessary, but they needed time to recollect their selves.
He couldn't let them go away and risk his plans. That was why Chu Fan was here, trying to put up the front of being at the café dealing with the issues in here.
But the show Tianlong Yun was trying to show, had Chu Fan as a main character, so the one who needed to show up at 'The Mansion' was Chu Fan and not him.