The gems just alter various stats. These are stats that are of value to the heroes who frequent the tavern. The prices they pay depend on them. So, if you win a heart gem, it boost the health stat of the food and drink you sell. This would mean more money for items high in health.
I didn't have time to fully incorporate what I had planned out. I was going to have a meter which showed the demand for each of the stats. This demand would vary over time, much like in a game like Sim City. The player would then have to balance stats via which gems were active (another missed feature). It would also give the player incentive to play the Blackjack game.