An interesting concept!
There are some issues if you switch between mouse/keyboard and gamepad, it seems like when one input is used it 'locks' that to the specific device. For example, pushing Y on the gamepad meant that trying to use Enter to finish the level wouldn't respond. (Maybe it's just me though!) I three starred both levels once I stopped trying to use a controller. :)
Congratulations on submitting a game to a jam for the first time! I hope you'll enter more of them in the future!