Not a bad idea to combine the theme with change of controls.
But primarly its sortoff undoing the fun you could have. Generaly these mindbending things are what frustrates people about games rather than the opposite. Making a game based on this is... bold I'd say.
Also. Given the cooldowns on potions you are encouraged to chug them as fast as possible.
People like me reach for WSAD first. And its 100% not optimal to use WSAD when you have to mash numbers almost as often. I'd suggest disabling WSAD and keep it on the Arrows cause its otherwise mostly unplayable.
Overall nicely put together, it runs alright despite the initial freezing.