He needs to have a certain amount of like i believe, I just spammed his workout event in the temple and did every event I could with him to raise it as high as I could. I also raised Othra's Rhot's Grushk's and Xer's too just to be safe. pretty sure I maxed them out as far as i could back then.