the only way is to clear all goblins and destroy all traps before talking to the girls, to avoid them hindering you. if get 60+ weak, some special goblins will become neutral
oh jeez. yeah i just started out basically, i cleared an easy path for them to all wander out, but since they pathfind like shit i basically had to save load every time they took the correct direction instead of going back or sitting still