what kind of controller are you using? and is it windows or mac?
I am having this same problem on Mac with Switch Pro Controller and Joy-Cons.
