Perhaps there can be some puzzles that are less reliant on character events? If it's really a no-go, you could just try changing the setup somewhat, maybe with illustrated cutscenes for events instead of the characters moving. I really do think that the idea is neat and it would be fun to see it expanded even to just one more level, but I can definitely understand the frustration.