Casually opening a hole in the wall is not something I would do. It's something that Kaisar has done before to access this place since opening the massive slab of a door is going to be an impossible task as that is designed to not open by any means and is made with the intent of sealing anything behind it.
It would have been fine if the walls were also strong but compared to the walls of the cave and the material the door made from, it's something that cannot be compared equally. Before, when the door was installed, the walls were as strong as the door, so no one is afraid that anyone will breach the walls. However, compared to the door, the wall is subject to the passage of time and it does not escape the wear and tear most things go through as time goes on, the walls would also lose the little strength that holds them together.