Old Pan's intention of inviting for dining, first, was to keep Yan Fei from leaving, and second, was to find out to what extent Yan Fei planned to handle this matter.
Yan Fei had always been straightforward and told Old Pan clearly that, anyway, the chief culprit causing the disturbance must not be let go. As for others, he would not care -- not that he didn't care, but that he would caution them a little bit.
After all, he found out by asking those children that not everyone in the village supported this event. Judging from those snotty kids, those who said Uncle Shuancheng was bad must have heard from their families.
That is to say, these households also felt that the idea of blocking the door was too damaging and somewhat inconsistent with the concept of villagers - since this was the case, it would not be appropriate for him to make too much fuss.