As usual when fighting summoners, you want to target the summoner, not the summons! When you fight a necromancer, you don't want to waste your time and energy beating up every zombie before dealing with their mistress; by the time you manage to deal with one wave, she'll be about ready to call in the next! Same deal with shadowy ghasts, except more so.
As for actually defeating them, magic weapons are important when fighting ghosts! Make sure you have one equipped, or your hit chance will be very bad. Also, this is the sort of enemy that doesn't like to close into melee range, so consider using ranged attacks. Also also, consider taking one of the perks that lets you see the resistances and weaknesses of enemies; some attack types are totally useless against some enemies, and you really don't want to be fighting an enemy with a weapon they're immune to!
If you aren't equipped to deal with ghosts, and shadowy ghasts are starting to spawn and give you trouble, think twice before draining mana from shrines! And if you aren't able to deal with large numbers of shadow enemies backed by multiple types of summoner, consider passing on the quest to fight the Shadow Clan.