Maybe you missed the dialog that told you to go back and talk with Aiyana about the last gem, after you found the previous 5 gems?

Location of all gemstones for Aiyana's Summoning Blade quest:
1. Tajurata Fort -> Second floor -> Shelf in the top left. (unlocks gem 3)
2. Jewelry Store in Thremten (unlocks gem 4)
3. Magic store in Witton (unlocks gem 5)
4. Yin Tower otuside (Fight against some bandits)
5. Diminus Castle (Guy in a small house)
6. House of the priester in Old Ashton (requires that you have all 5 other gems first and then talk with Aiyana.)