Someone from this forum figured it out.
https://harem-battle.club/general-discussion/5254-town-sins-game-16.html
Basically the game only freezes if a card with butt plug ability attacks and there is no opponent cards left on board. I verified it and I'm able to work around the freezing issue for most of the battles.
It has nothing to do with game speed, but this also means avoid using auto combat.