I like concept, but the color-coding by itself isn't enough. The shapes of the notes should reflect the arrow key you're supposed to use.
Also, very unusual for a rhythm game, it seems too slow. There just aren't that many beats you play. It ends up making for a lot of waiting and a weirdly slow handjob.