"Earl, you have finally returned; we were so worried. This time you left without taking us, and we've been anxiously awaiting your return day after day."
"Indeed, Earl! Next time you go out, you must take us with you. Otherwise, it makes us worry terribly and we can't offer you any help."
One Viscount after another expressed their concerns, which in reality were a pretext to go outside. The Earl had slipped out quietly this time, and by the time they found out, it was too late to follow.
The family prohibition kept them from leaving. The Earl could still ensure a measure of their safety. If they went out on their own, they would definitely be in danger.
The Earl had used his magic wand to avert danger during their past few risky encounters.
The Earl said to his brothers, "Following me doesn't guarantee your safety, and this trip was too dangerous. Someone was tracking me, but fortunately, I had the magic wand to remove the threat, which is why there was a day's delay."