It wasn't like he had a grudge on her or anything. Instead, he simply didn't wish to get involved in the continental war. After all, the lady that had called for his help was a princess of the Zamuria Empire.
So, the moment he helped her, he would be making an enemy of the Guria Empire. With that in mind, that was just an unnecessary headache for the current him that was worrying about being able to find an infant continent.
That was why he hadn't attacked anyone all this time, only retaliating when they came at him with the intention to kill. On top of that, to avoid killing people unnecessarily, he showed his power first, making people hesitant to attack him.
His goal was to safely reach the Guria Continent and then head beyond. With that, he didn't wish to become involved in anything that might cause him difficulties later.
Unfortunately for him, reality proved different.