Thanks a bunch for the comment! We agree that the limitation wasn't implemented thoroughly. There were so many things planned for the rhythm mechanic that ended up getting scrapped. Sorry about the difficulty too, we would've loved for you to have played all the levels! We hope you enjoyed what you were able to play of the game.
Thank you for the feedback and thank you for playing!