I like the idea. I think it definitely does ramp up in difficulty a little too fast. Starting out with some dead obvious levels would be good just so the player can get used to things a bit before it gets hard. It was also difficult to remember based on just shapes and colors alone. Swapping out that for something more familiar (like objects or people) could make it feel more natural.