Units sometimes go through buildings (tested with warbands and migrants). I started this game yesterday and they didn't do this, they only began doing it after the game had been saved and loaded.
gif of this happening: https://ovnize.s-ul.eu/vDWmADn... (I don't think it is related to the gate being close to where they did it, I've seen it happen with a lot of other buildings)