If you don't get enough gems on a given level, the next level becomes impossible - this seems a major flaw to me. However, I think there's a seed of an idea that could be interesting if explored further. Perhaps than losing basic movement controls, the gems give you a range of different power ups that make the next level easier, but still fundamentally possible even without collecting the gems. Though, I'm not sure if that would fit the Game Jam theme.