I thought it was just me who had this bug! I think it has something to do with the jam pages themselves which is very confusing, but I tested the same upload to a regular itch page and it didn't have the mouse drifting issue.
The good news is playing the game in full screen mode doesn't have the issue.