"At least four major opportunities were missed, which resulted in such an unfounded accusation that led to my death." Pei Xuanjing thought with displeasure.
Firstly, at the time of his rise at the frontier, he had enough time, which meant that he must join the contest.
In struggles, there is no place for fence-sitters. Especially in a fight over the imperial throne, where it's a deadly fight of winning or perishing, his decision to be an observer would make him the common enemy of the crown prince and many other princes.
Also, when the old emperor borrowed him to balance the many princes and the crown prince, even if he genuinely chose to be a bystander, he could seize the opportunity to strengthen his influence, ensuring himself to have enough self-protection power.