I waited for a few hours until the guards changed shift. It's about time that they showed opening.
I think that because this city has never been attacked ever since the cult started residing in this city, no one taking the guard duty seriously. But they are still expert level mages and master level mages. So even if they seem relaxed, they can sense a bit of change in the surrounding.
If an uninvited guest suddenly appears, even if the guards seem to be relaxing, they would notice. At least the master level mage among the group will. And that mage will report it to the others.
And there are also almost no opening at all in the spots they are watching. Other than the opening to the prison.
The castle is big, so even if it's in the center of the city, I can still see a part of it. And my Divine Vision is good enough to see up to the prison.