This feels like a neat idea! Unfortunately the game jam time constraints seem to have hurt the fun a bit, a smudge more gameplay/levels could really gone a long way for this.
Nevertheless, well done!