I just gave a try to fullscreen mode and there's absolutely no slowdown, it's always very smooth and fast even when there's a lot of floating items, .so it seems the problem is indeed the windowed rendering .
If the slowdown occurs specifically while the dice are in motion that could very well be the case. In any case, I'm working on some performance improvements (including a vsync toggle) for the next patch!