There was a clear distinction between the six. The Sacred Federation warriors being led by Alps Mohan were obviously unwilling to get anywhere near Xia Fan, and when they continued their way forward, Alps MOhan and the others were at the front with Sarah, while Xia Fan kept a careful distance behind them.
Xia Fan did not really mind being in such an awkward situation. He had studied medicine from Doctor Grandpa, and he knew that Sarah's condition would stabilize in another day or two. She would be able to walk just fine when that time came, and he would then be at ease to go searching for Traveling Buddha and Thousand Ink when that happened.
Meanwhile, leaving now would make him feel as if he had abandoned all his efforts. After all, Xia Fan had already spent an entire week taking care of Sarah, and since he had decided to be a good person, he might as well see it to the end.