I agree with that, it is actually a bit frustrating for me too, but it was the idea. These kind of situations are all about patience ^^'
We though about it more like an interactive experience than an actual game, tho it'd be nice to make a more "gaming" version, more fun to play.
Thanks for your feedback, it's really useful! <3