Woah that's a really creative concept. The aesthetic is nice. It took me a long time to work out what to do, and to be honest I don't think I was 100% certain that I understood how it worked, I found it very cryptic. Nonetheless I completed the game only skipping one puzzle.
I agree with the other comments that a fast-forward or skip of the music part would be nice.