This was really clever! and reeeeeaaaally hard! I was very close to giving up, but I think that that is actually a good thing!
Using a touch or pen input would definitely help.