Most of the Prisoners that were freed didn't work together. Even though they had a common enemy, they were still part of different factions.
Most of the independent prisoners rushed out of the Cellar Area without caring about others a single bit. They didn't care if others were free or not. More than that, they were worried about the Queen who might return.
Even though they hated her to the core, they knew that they couldn't do anything. They left that for others to handle and escaped.
On the other hand, the upright Warriors from Various Abolished Kingdoms stayed behind. Unlike the ones who escaped, they were wiser. They knew that strength was in numbers. The only way they could survive was if they freed every single Prisoner and worked together.