They don't, It's just that they spawn when you kill enemies, and when you're focusing the golem nothing is dying. You should try save some big potions to deal with each golem.
