Oof yeah that's a bug folks have been running into a lot. I wanted to work on a fix but toward the end of the jam we just didn't have time. Totally will be a priority if there's an extended non-jam version of this game.
