Since sound is currently the main thing through which you get feedback on the accuracy of your timing, playing without makes it harder to win.
The best moment to hit a key is when you are about 3/4 on the way to a square, once for each sqaure that you move. The swimming animation will more clearly hint at this moment in future versions.