Just move the app in any other folder and it should fix it.
Don't ask me why, I tried a lot of things, it's the only thing that worked. It's a known issue apparently.
I fixed this as follows:
xattr -d com.apple.quarantine Exterminator.app/
