Hello there, first of all, thank you for your support. :)
The only way to get rid of the sad debuff is to simply wait it out. Since it's inevitable to get the sad debuff when the Criers (The talking zombie) appear, you should consider WHEN you take them out.