I feels like there are need some fix for controller/gamepad option.
switch mode button mapping not yet existed in the gamepad setting. But if you press LB button, its double input for switch mode and inventory.
I still haven't fully set up the buttons on the controller/gamepad. I'll try to fix it in the next update. I apologize for the trouble you're having.