Upon living the life of someone he wasn't supposed to be, Desmond gave up on having friends. The only people who knew who he was were the king and Gideon. He preferred to keep his circle small rather than pretending he wasn't someone more grand than he was to those who didn't know the situation.
Despite being born a noble because of his father's military career, his formative years were in an orphanage. He felt socially inept in the face of other nobles. Luckily, his appearance spoke for him and none desired to get close to him anyway.
However, commoners were much harder to handle. They didn't care who he was, or what he looked like. He was the people's hero, not the nobles' hero. They thought he was honorable, unlike the nobles.
When he appeared in front of them, especially the people of Montclair, they gave him respect and didn't mind his presence.