Okay, I think I understand what's going on. There is a lot of lag on many things in the game (including the cursor which is a custom cursor).
This is certainly caused by the numerous effects (particles, multiple interpolations on each brick and on each leaf, several CRT and Glitch filters, etc) which require a lot of CPU and graphics card resources.
If I were to make a more complete version of the game, I'd certainly add a menu to activate/deactivate these options, as well as a message warning of possible lag. In the meantime, you can always try unchecking the “Filter” box at the bottom left of the screen. I don't know if this will be enough, but it should certainly cost less resources.
Thanks for taking the time to make this video and for telling me about the problem :)