It might be with the order of your ray casts. You are performing two separate ray casts, one for reflection and one for checking gem collisions, which could cause this. Instead, you should combine the reflection check and gem collision check into a single ray cast.