Oh, that's not implemented yet, at least in the public version. Hyao said at a moment that, if bandit's moral is low, the can attack your house.
I do have the latest version and I tried to plummet bandits' morale and reputation as low as I could make it but nothing occurred so...not sure how exactly to trigger that.
