If you don't like buttons, don't play this game.
It feels a bit too trial and error-like, maybe give the player a way to discover the 'correct' button