Have you considered any alternate control schemes? I just lost a 77 streak because the buttons decided that I briefly released my hold, I guess...
If I could offer a suggestion: it could be something like, holding A and pressing a direction shifts the row/column in that direction, and holding B and pressing a direction shifts the colors? (I know it still involves holding, but it's in such a way that if the button releases, nothing happens, rather than it committing an action.)