Except for the first greetings, there was no need for all of the agents and escorts to be around all of the time. That would be too cramped and would instead negatively affect their job.
Plus, they also had to rest. They were humans so they couldn't just stay guard all of the day. They couldn't protect the prince and fight properly if they were tired, could they?
That said, they had two working shifts set for most of the days. Except for the times when the prince would move out of the city, which all of the agents had to be present to protect him like when he would visit the borders or some meetings that would take place in other cities, there was no need for all of the agents to be present because SMF obviously didn't want to kill its own agents by overworking them.
