The Prisoners that were in their houses all came out as they felt the ground tremble. They have not really experienced an earthquake before in the Pit, and so the looks of fear on their face could be seen clearly.
There's also the slight fear of the Locals. What if that rumbling was from them marching their army inside the Wall? They do not even know how many there are, but seeing as they have more than a thousand years of history and a thousand years to populate this place, there should be almost more than hundreds of thousands of them. Even a million if they weren't in the constant threat of the monsters from the Portal.
As far as the Prisoner population knows, the only person alive today that has gone to the other side of the Wall was Boss. She was also known to be the oldest resident of the Pit now. Some have come and gone, and yet she was still here.
"What… what was that!?"