If they had betrayed their employers before, they could do it again, so Castor wouldn't have them around where they could be hurt while Andrew was away, and he was even planning to send some of the Shadow agents with the caravan to monitor them.
Certainly, these agents couldn't keep these experts at bay, since they were weaker, but they could still report if any problems occurred and the experts didn't show up.
This way, they would know that they had fled, and if instead they became more aggressive and killed the Shadow agents, well, the situation would be even simpler, since this is easier to find out.
While there were many ways in which these experts could play their cards to escape, the problem was that they didn't know the extent of Andrew and his collaborators' forces and influence.