Are they in the middle of a street, or in a cramped area? Could be a bug with formation movement. You can order them to garrison in a settlement, that should cause them to break formation and move.
I'm afraid that could be related to a bug around warbands getting stuck while pathing. An enemy warband wants the center of its formation to be in your town, but the warriors can't actually get there. Your warband is trying to fight the enemy, and is squaring off with where the enemy wants to be.
Unfortunately there's no fix for that yet, sorry about that.
A potential workaround would be to garrison that warband in another settlement, which would put them out of aggro range of the enemy warband's supposed position.