What Golver implied with his words was that although he didn't know who the man was, he was sure to be stronger than any of them individually.
Since none of them had any enmity with him and he expressed his intentions to not contend for the legacy since the beginning, they were able to trust him to some extent.
This made it so that Hans and Danulo unconsciously closed the distance between them as they prepared to hold the man off in case he sneak-attacked them.
In fact, it wasn't just them who were surprised by Golver's warning. The Harita ancestor himself was surprised by that claim.
He was certain that no one from the other two tribes could have recognized him due to the curse that he placed on the bloodline of the Harita tribe. If any one of them tried to betray him, their death would only be miserable to look at and he would also be aware of the betrayal.
