There might be several reasons why the trolls were able to stand against the human's siege way better compared to the ogres, but two of the most crucial points that enabled them to do so were their earlier knowledge of the human's ability to cast magic spells from Matthew and the trolls were also capable of casting their own magic, which was earth magic that enabled them to make three-meter tall, tens of meters long walls in just one minute.
Both of these reasons gave the trolls the edge they needed to be able to counter the human's move while making their own move forward.
With them being aware of the human's capabilities of casting magic, Trolgar was able to prepare beforehand and readied himself to receive magic attacks from them from the first moment the humans were seen closing in on the troll's village.