Hmmm.... the use of Alt as a control seems to introduce a lot of friction with my Window Manager. I get stuck going a direction even after letting off all the controls. I've seen this happen trying to play DOS games using DOSBox when I haven't rebound Alt, too, so it's definitely something that happens consistently to me.
Unsure if this is a problem for other people (using XFCE in Linux is probably relevant here), or if it's just my untypical setup.
(Found a workaround -- it seems z also works as an interact button)