So, if I left click inside the game window and the cursor exits this window the input counts as released. If I could fullscreen the game that would be fixed.