LOL!, if you go back to the gem's original location , a dialog will ask if you want to return it :)
If you watch Ecaroh.games' playthrough in the comments, the player didn't get the popup to return the gem because he hadn't left the gem trigger and reentered. The gem trigger is about 3 times larger than the gem and the dialog (take gem or return gem) is called on OnTriggerEnter only (not OnTrigger or OnTriggerExit).