The old wizards were already worried about Oliver's presence because although they were all at the super expert level, it was obvious that Oliver was more powerful than any of them. However, they were still confident since they were 5 against 1.
However, the appearance of an expert even more powerful than him made them nervous. If they added to that the presence of Castor, who, although he is considerably weaker, they could not disregard him or they would be in danger.
They still thought they could handle the three of them, but they were sure that it would be a difficult battle and that they would probably suffer a heavy loss even if they won the battle, so that was why they now were reevaluating the situation.
Now more than ever they wanted to know what had happened so that a force like the three in front of them would appear to attack them. However, at that time the situation changed again.