Maybe that "Fast Dice" by cutting down the dice jumping and bouncing is somehow lowering the heavy load of whatever is causing the slowdown ?
If the slowdown occurs specifically while the dice are in motion that could very well be the case. In any case, I'm working on some performance improvements (including a vsync toggle) for the next patch!