Although Andrew and the others, even Fluffy, had doubts about what Elidyr had said, the reality was that the old man was not lying to them. The reason they doubted his intentions was that they could not understand the reason why he would give them something so valuable.
It is true that Elidyr had told them that they had the inheritance of their ancestral spirits and that was why he could hand over the remains of the deceased gods that they had in their possession, but even for Fluffy, who is a divine beast, it was difficult to believe that they would give up that power just because they wanted to honor their heritage.
The problem was that they could not understand that for a person like Elidyr who had personally experienced the war against the fallen god a millennium ago, the vision of the situation was different.