No one would consider the phrase "selflessly impartial" as derogatory, nor would anyone think it describes something undesirable.
Of course, that is based on the assumption that one would not be placed at a disadvantage by it. Once at a disadvantage, people no longer see it that way; those who deal with others based on selfless impartiality are often considered "indifferent," "heartless," or "mentally troubled."
Take a simple example: while people like incorruptible officials, very few would be willing to be relatives of such an official. That's because people detest corrupt bureaucrats, but they also dislike relatives who are officials and who are unable to seek benefits or conveniences for them.
Such is human nature. It's unrealistic to expect everyone to be a Sage, to treat everyone including themselves with absolute objectivity and fairness.
