From what I can remember,
For Ferro, if you do his sex scene and submit to him each time, he'll gain a rank in a ability that makes him grow after you win any encounter.
And, for the imp, if you completely dominate him, you can grow him with essence when you check on him while he's your companion. If you fully submit to him, though he'll grow after you win or lose to him.
As for the other encounters, yeah, sadly that's the only way you can grow them.
Edit:
I forgot Ethrex, you can grow him if you lose against him when sparring with him while he's your partner.