Being brought to a new and unfamiliar place during the night was not what Waker expected from his life. However, they had moved as a party in the night before, so that did not make this the strangest traveling plan that they had.
He understood why this was important. With how fast they could travel as just the pair of them, they could do what it would take an army to do in hours versus days. That was just part of the benefits here. Walker knew he would be able to begin exploring right away with the time of day.
The crystal islands were dangerous during the day, getting there at that time would be a limiter for him. While Walker knew that he could in fact resist the light and the heat, he would prefer not to have to work on that constantly. The less mana he used there the more he could use for other things.