Indeed, they were 'invited' to come.
Because these people were originally ruffians and criminals recruited by the Nangong Immortal Family from the northern states to do evil deeds that rendered them incapable of finding even a place to live in the north, only to be driven to desperation in the Southern Frontier.
With the Immortal Palace's twelve guardian families, and with the Nangong Family's support from behind, naturally, no one could control these demons.
So what did the Nangong Family hope to gain by wining over such fiendish heretics?