Although no one believed that the royal family would mistreat them, everyone had thoughts similar to Oliver's. They thought that something bad could happen and that it was better to safeguard their safety.
So far the royal family had behaved wonderfully towards them, but recently the release of Andrew's divine seal brought some uncertainty and now they were in a fight with one of the nobles of higher status and seniority, so being worried was normal.
However, Victoria intervened, "I have those worries too, but I can tell you for sure, if they wanted to do something to him, they wouldn't have called me. I know them very well, they would never use me in that way."
Despite Victoria's comment, everyone felt insecure, but still, Andrew agreed and left with Victoria for the royal castle of Dunnas. Oliver and the others couldn't help but worry, but since Andrew had made up his mind, they couldn't do anything.