Diminus Castle unlocks after you found the strone in Witton, which unlocks after the stone in Tajurata Fort.
Location of all gemstones for Aiyana's Summoning Blade quest:
1. Tajurata Fort -> Second floor -> Shelf in the top left. (unlocks gem 3)
2. Jewelry Store in Thremten (unlocks gem 4)
3. Magic store in Witton (unlocks gem 5)
4. Yin Tower otuside (Fight against some bandits)
5. Diminus Castle (Guy in a small house)
6. House of the priester in Old Ashton (requires that you have all 5 other gems first and then talk with Aiyana.)