Try to attack repeatedly or use multiple his attacks to chip the barriers.
Actually I just save scummed my way to victory. I guess it was just an RNG issue. Even then, once I got past that hurdle, everything else was alright.
