I believe you either build their trust stat or affection. Hard to remember which since you generally get both if you act reasonably. I'm guessing trust though.
