Zafir had sensed that he and Stella were being watched every time they went out so he had asked some of their elite knights to secretly follow them dressed as normal people.
He didn't feel any malice towards Stella so he at least knew that they were not out to harm her. But they may still be in their planning stage so they still had to be cautious.
He surmised that it was someone from the temple so he asked Agatha and Camilla if Stella had any enemies in there. There might be people feeling envious of Stella's special treatment.
Surprisingly, Stella had friction with the Grand Magus' nieces although they have never met in person yet. He grimaced when he learned that it was due to the three princes vying for her hand in marriage.
The elite knights didn't notice anyone following them so they surmised that they were spied on through magic. Since it was probably from the Grand Magus family and their supporters, it was highly likely that magic was involved.