From what the three brothers had learned from the man, it seems that the towns in that place had existed for years now. That the people living in those towns had existed ever since the beginning and that the reason why no one had discovered them was exactly because of their location.
The Prism Heart was protected by natural barriers and there were beasts scattered about that will attack anyone that would step foot inside the territory. Apparently, this was the first time that a group of outsiders were able to bypass those barriers and defeat those beasts.
The man was dragged away and was dumped together with the others. He was told that they will talk once again once they organize the information they got. The brothers shamelessly occupied the leader's manor since they believed that they will soon become the masters of the town so there was nothing wrong with them using the manor.