Really cool concept!
I believe some sort of tooltip telling you to seek a gem at the start would go a long way in improving the experience, because when i started playing i wasn't sure what my goal was.
Other than that I had a lot of fun, and the level design and execution is great!