The mini-game during combat takes into account your flexibility and your opponent's. The worse your flexibility is, the less time you have to press the button. This difference also affects the size of the button. Perhaps the problem is that you do not hit the button accurately, or do not have time to do so.