Sam isn't the type to make a fuss for no reason, nor is he a pushover.
He certainly doesn't like others to think he's easy to manipulate, always good-natured and never angry. His attitude depends entirely on how others treat him. And that inexplicable man earlier, Sam didn't have a good impression of him at all.
True politeness involves treating each other as equals and engaging in friendly conversation, not looking down from a high position, dictating terms while smiling.
His politeness was a kind of charity, as if asking Sam a question offered Sam the opportunity to engage with someone of his stature, and if Sam responded, it was as if he was being shown a sign of respect.
Moreover, he was here looking for Zoe... this left no chance for Sam to feel any goodwill towards him.