Really interesting and clever game mechanic. Also, I think that this mechanic could potentially be extended into other things as well (perhaps puzzles) which would make the game even more interesting.
I do think that the character movement needs a bit of improvement as it doesn't feel particularly smooth. Also, the ladder colliders and movement could use some improvements as well.
Other than that though, this was a great game!