In that "Phantom Tower only visible to those who yearn," there was an abundance of deception.
And to deceive others better, one must first deceive oneself.
Thus, the apprentices in the Phantom Tower could, through a simple idol spell, excise parts of their memories and personalities and turn them into objects to hide away.
This spell could even be used by those who had not yet advanced to transcendent status. The upper limit of "facets" one could hold at a time was, however, closely related to one's own strength.
By cutting parts of themselves away to a certain degree, they could precisely and controllably alter their personalities and thought processes.
For example, 70% of fear, 90% of rage, or 50% of greed, and so forth.
In this way, even if others were to pry into their memories, no flaws would be revealed, and when others tested their temperaments, they could very naturally demonstrate their own harmlessness.