You also need to get their affinity up, by talking or giving them gifts at the fruit bowls in Inns
Thank you! <3