One time when I assaulted an enemy settlement, wiped out its defences and had it separated from incoming reinforcements leaving only one infantry unit in the settlement then and I surrounded it at all 4 sides and I attacked the settlement but despite all the supporting units supporting the unit I used to attack the settlement the red unit in the settlement still managed to make my attacking unit to retreat and that gave enough time for more reinforcements to flood the site. Is this bad luck or is it because of my foolish choice/s like not to use my reserves?
I would consider outcome not improbable. Let's say you have a settlement surrounded by 4 infantries and the settlement is defended by a single infantry. If now one of your infantries attack, the odds on your side are attack 3 + 3*1 support = 6 against 4 defense *2 terrain = 8 . So your success probability is only 43%! If different unit types or counts are involved, the individual numbers need to be adjusted, but generally infantries within settlements make formidable strong points. Unless you manage to provide some artilleries to support your assault from behind...