Boost bandit's attitude and moral towards you. The easiest way - to talk to Jester and give money to the bandits - 2 talks per day are possible. 2-3 days and you will have calm nights
Thanks Heaven_Steed. I paid Jester money for bandits and that helped me progress further.
