If you play the newest update 22.06.03, the timer for crystals to re-appear increases slightly each time you use the button. Therefore you should be able to do it eventually, if you try often enough.
You need to put at least 5 "Strange Fairy Dust" into the flame in the center of that island to make the treasure in the hole in the north obtainable. (All dust you have your inventory is removed when you put it into the flame, even if you have way more than 5.)
