Your idea is actually very nice, I think I will take it in consideration as it can add the puzzle flavor I would have loved to put in tehe game, but did not had time.
Yeah the gravity is applied by me in the update to a character controller, did not use rigidbody for it and it does not stop multiplying if you are on ground so when you fall, there is a very big number for the y velocity =))))
Would have been a quick fix, but forgot about it totally
=)))