The garden bonus is due to the slave being in a "fresh" location. Using this about four or five(?) times in a row will cause it to no longer give that bonus. The streets and bedroom also give bonuses if it's the first "fresh" location you switch to in the interaction.
Viewing post in Old Bug Report Thread
Pretty sure there's a bug in this case... since I been doing non-sex interaction at least over a hundred times. Often 3 times in a row for the same character in one day and multiple days in a row. It always give +5 action turns.
There also doesn't seem to be a check to make a character refuse intimacy since I can literally have a newly enslaved character accept my intimacy proposal on the first try where both their obedience and loyalty are in the red zone.