Abilities weren't simply weapons, they were also part of who a person was.
Even a simple weapon, however, when used over a long period of time, would have the ability to influence the wielder, especially all the more so when the weapon in question was a person's own Spiritual Ability.
To state it in more extreme terms, it was that power had the ability to change people.
Perhaps some might find that statement ludicrous, but this statement was in fact indisputable.
Perhaps weapons and abilities were bad analogies, but if concepts such as wealth or authority were considered "powers" as well, it was quite normal that such "powers" would change people's personalities. In fact, it would be quite common.
Seiji had many experiences with this in his previous life.
That was why he restrained himself as much as possible from abusing his own incredibly powerful ability of saving and loading.