Andrew didn't have to be present to know what his wives wanted to talk to Fluffy about. She had declared herself as one of his wives and Andrew didn't deny it, so the deal was practically sealed.
Now as a member of the harem and as the closest person to Andrew, having lived with him sharing mind and soul for several years, it was obvious that the girls had much to discuss.
Andrew couldn't interfere in that, so he simply let them do what they wanted and he focused on talking to his guardians about the meeting they would have the next day with the demigods.
Although everything had been discussed with Crhono, some things still needed to be planned. Andrew didn't plan on meeting these demigods accompanied by their wives and guardians.