Managed to beat the game doing what you suggested. In a way, it kinda works more like a puzzle game than a rpg, which could lead to some interesting design opportunities.
