I liked a lot your take on the concept. However, since some objects aren't controllable in some levels while others are, could be helpful if there were some indicators to it.
Since I smartn't, didn't manage to finish it all the way thru - but pretty neat.
Edit: On a second note, affected objects could be a bit more consistent between levels - Since it's weird that in some levels the flag isn't controllable and that the score is in others.