There were several requirements that people had to meet before they could get this job. First, they must be male; second, they must be strong; third, they must have broad shoulders.
Why did these people have to have broad shoulders to get the job? It was because when they did the job, they needed to use their shoulders a lot, and having broad shoulders was like proof that they had years of experience doing similar work before they came applying for the job at the Prince's quarter.
Yes, it was the Prince who employed people who had to use their broad shoulders and strong hands to hold the palanquin. To be precise, it was the Second Prince of Si Long who hired those people to help his palanquin move from place to place.
There were indeed some requirements the men had to meet, and even if he carefully selected those men before they could finally land a job as palanquin holders, he still felt uncomfortable traveling on a palanquin.