It was the Book of Heaven's Path, where he had sealed Vicious within!
Vicious had once told him that the other Vicious was his brain and his eyes, and the innate authority the brain possessed allowed it to exert dominance over the other body parts. As such, it would have been impossible for him to possess the other party under normal circumstances.
The only way he could successfully possess the other Vicious was when the other party's consciousness was in a state of confusion, rendering the other party unable to counterattack his possession!
Considering how the other Vicious was a soul oracle who possessed the intact memories of the original Vicious, it could be said that it would be nigh impossible to place his consciousness into a state of confusion. However, upon learning that the little island was an artifact that the other Vicious was intending to use to house its will, the rest was easy.