Yeah, difficulty was a struggle to balance. I spent pretty much the entire 4th day of the jam playtesting and tweaking things, and I still don't think I got it quite right. If I were to continue developing this game after the jam, as unlikely as that is, I think I'd try to add difficulty settings.
I think that it is very difficult to tweak difficulty all by oneself. One become too deep in the project and cannot see the whole picure most of the time!