The pirates that Stitches encountered during his transport had already been enslaved and died in battle, but that didn't mean there wouldn't be any pirates this time. There was no way he would be able to escape from the military while at the war camp. The only possible option for escape would be during the transport to Beltor's Platform. But Stitches couldn't pull off the escape alone, so he started gathering like-minded people.
Considering the circumstances and what would happen to the people on the boat headed for Beltor, few refused Stitches' suggestion of a prison break.