After Dantalian's conversation with the villagers, it was only a matter of time before the rumors were shattered and everyone began to ignore them.
Dantalian never actually wanted to tarnish the Duchess's name. After all, how could he use her influence or, worse, what if his invitation was invalidated? Absolutely not, he didn't want that.
He decided to tell only a partial truth, albeit a milder version, among various other well-crafted but exaggerated lies that would serve as blatant disguises.
By doing this, the fact that one of the rumors turned out to be true could easily be attributed to sheer luck on the part of the true villain. Yes, the person who spread all the false rumors and now drew all the attention to themselves.
No one would be foolish enough to appear, right? And so, the problem that would gradually lead Duchess Marian to ruin came to an end.