Adrian knew that he cannot change the minds of the three riders unless he finds the soul of the last one. It seems that they are very connected with each other as they are literally foregoing the chance of resurrection unless all of them are present. He does not know where the confidence of the three are based on, but he does not want all this expectation presented to him.
Sometimes heavy expectations become immense burdens for people as they cannot fulfill such an easy promise or do what they said they would do. You can also let a lot of people down if you do not exceed it and plunge yourself in agony and despair.
