No, it doesn't slow the timer down, but it's much faster to repeatedly hit Enter/Spacebar than it is to move the mouse and click on the button. It's not a big problem, but I just wanted to make you aware.
I could call that a feature for people who find moving the mouse too fun.
