After she discovered who the current Queen of the Sound Kingdom was, she felt a lot more relaxed being in a foreign country. One of the main reasons why she remained in her room other than her unstable mental state was this. She could hear the gossiping whenever she opened the window just a little bit.
Due to the orders of the Queen there are not many servants assigned to this area she stayed in. Chisa must have picked up that her people didn't like her. The main reason for their dislike is, because of how Kazuya treats her.
Kazuya is unable to hide the feelings he has for her, he is very obvious. The servants are under the delusion that she came back to take Kazuya away from Chisa. 'Just a normal member of the Sound Kingdom?' The people here have long been treating him as the King.