The idea works well! Sadly, I think I softlocked because the red square is stuck...
With more time, you could turn it into something really cool!