This is a neat idea but nothing stops the player from mashing every button to instantly clear every distraction, trivializing the difficulty.
That's a really good point, I'll keep it in mind. Thank you!