As others have said, the controls definitely need to be a bit more traditional, and the sensitivity made things a little difficult.
Other than that, though, I liked the idea! Loved how things kept changing, it felt really trippy - and the replay at the end, showing what really happened, was a very cool touch and I'm sure it took a while to get right!
A welcome addition would be the ability to speed up the replay segment. Add some general polish and touch-ups, and I think you could have a very unique game!