That seems to be your problem, you should focus on debuffing the boss first. Any attack reduction on the boss is the same as a likewise defence increase on the entire party.
