Yes that's a known issue, I haven't managed to find a fix for it without changing the very core of the input system (which I am reluctant to do since that might cause other issues), so you will have to work around it by only have your controller connected.