Tarsh had taken the two spirits back before the other guys woke up. He didn't want them to see spirits and freak out. Neither did Dhvani and Ajay. So the now conscious group did not know that they had forgotten the most exciting journey they had been on. Ajay did not forget in the light of the system and of course, the spirit emperor Tarsh.
Because the world they were in was a time loop, and also the cave spirit did not want them to know this world, he had erased memories of the group. The cave spirit became the patron god of the newly created world and he of course had the power to do this.