Definitely happens with the mouse as well unfortunately, but I like the idea of a reverse roguelike - that's clever =)
I dug into it, and found the problem! The aim angle was defaulting to 0 every frame unless there was mouse input that frame. Thanks for confirming you were seeing the issue too, I've patched it now, so hopefully it's a better experience!
Well done, congratulations!
