Yeah it confused me too lol
You need to confirm talking with him multiple times, eventually it'll switch the buttons and that'll be the last time