He had already fretted over the possibilities so many times now. The chance that he couldn't return, the chance that even after returning, everyone had forgotten about him, or moved on with their lives, or had even died. The possibilities were endless, but at the moment, that wasn't what he was scared of. What he was scared of was confirmation.
Confirmation, be it from the Heavens, the d*mn voice, or himself, that his plan that he had painstakingly worked towards was unfeasible from the start. That he had put in all the blood, sweat, and tears for nothing, only to find out that this dimension at the dimension Earth were in didn't cross paths at all. Or that the cultivation technique couldn't find it.