the migrant groups were outside the settlement range, they were idling in the masklings camp where they spawn (unless they overlapped)
Also sometimes migrants will get stuck travelling to a structure that is just on the other side of something ex: carry bricks to a wall but they walk up to the tower just in front of it instead of going around through the gate
Does that make sense? I often have to delete the structure, and retry but it can happen again, it's almost like I'm guessing if this can be built or not in this place E. Happens almost always with walls in corners that sorta thing