Currently there are five world fragments, each in their own dungeon with an accompanying quest. As well as four world shards scattered across the world.
Could be a bug if one wasn't counted if you did four of the world fragment quests already. Luckely they currently serve no purpose so nothing can be missed.