The other two locations aren't necessary to advance the story. If you go there, you'll find that the fragments are already gone. The dialog with the King is the only way to advance the quest.
You can still visit those locations, if you want to do the riddles, but it's not necessary.