I suppose ghosts just move towards the player and only their colliders stop them from overlapping
Yes you are correct! Removing the colliders will make them stack on top of each other. The main problem is the basic AI, it needs improving.
