If you really look at it from a legal perspective, grandparents have no chance against mothers in a custody battle over a child; the law simply does not support them.
Not to mention that the Su Family had once done such despicable and heinous things to Lu Yun. If things really turned sour, and Xia Yuan decided to go to the police station to report them, they could not hope to come out on top.
In the end, both Lu Aiguo and Lu Yun wanted to avoid the Su Family not because they feared them, but because they were concerned about Xia Yuan's emotions.
As long as they had a few months in Haicheng to buffer, allowing Xia Yuan to step out of that shadow, by the time they returned to Yushu Village for the New Year, they would be completely fearless.