Really enjoy the concept for this game! It felt quite unique and really groovy to play as you got the hang of things :)
One thing I noticed was that I often didn't really feel like I was swinging to the beat or working in tandem with the music. I'm not sure how it would really feel, but perhaps having the notes fly over a dead zone that you have to either hit with the mouse or swing the red ball into would establish a more solid flow.
Regardless, I had a fun time playing this and think you made something pretty cool, good job!