I think this could be a really amazing game with some changes - This is only my opinion, and please do not take it as if I thought I know this a lot better than you do. I'm not trying to do that.
So, I think the most fun difficulty was "Difficult", because all the others had controls that change super fastly. But still, I think if you gave the player time to at least look at what changed would make a huge difference. Maybe splitting it up into waves, and the controls only change between waves, and so the player has a few seconds to look at the new controls. The current way, you have pretty much no time to look at it, since you would be dead almost instantly then.
I enjoyed the game nonetheless, I would just like to have a mode that is not this insane on control changes. Since this made me look like I suck at games even on easy difficulty. :D