Thanks for suggestions.
Bandits will play big role in the game. Pretty much everything you said is planned. You will even be able to even join the bandits. I plan to do proper faction system later this year, which will include basics of the bandit faction together with their hideout.
For starters, this / next month I could simply flesh out the existing bandit group and place it in more places as time goes on. I will add more checks to them, so they leave you be if you have certain number of followers with you, or if you kill enough of their friends. They will also leave you alone if you don't help the concubine. This should make them feel more like bandits and not just like wild animals with some dialogue.
And if I work fast enough, I might even be able to add the option for you to rob traveling merchants if certain conditions are met. I plan to do some kind of crime system in the future, so this should a nice and easy to create starting point.