A conductor rhythm game is such an interesting idea, and I really like the one button approach as an abstraction of that action, though playing with other cues and input is definitely a possibility. If you were to go down that road, I'd just be careful to avoid keyboard-ish cue designs. Other than that, I think you've already mostly identify where there's room for improvement (though I do quite like the visuals already!).
My main other thought is that you may be able to play into the role of a conductor more in how they control the pace of the music. For example, maybe some cues have larger timing windows and depending on if you play sooner or later in those windows the music speeds up or slows down, or maybe some cues simply pause the music and wait for play input like this level in Rhythm Doctor on "so long." It's definitely a technical challenge, but would be very unique for a rhythm game and really represent the player as a conductor.