Byron wanted to hit himself. Talking about the treasure grove was still safe. But did he have to blurt out the treasure grove was connected to the Void Master? The man in black killed the people who ambushed them, which meant they were not on the same side, but his identity was still unknown. He could be an agent from another group planning to hinder their plans. And him blurting out the information about the treasure grove was practically sentencing himself and the Blue Lotus to death!
'I'm sorry, boss. I failed you!' He thought in frustration.