There's a secret tunnel exit under the entrance to the troglodyte lair. It looks like a bunch of solid squares. Pass by it back and forth until you find the secret tunnel. The next area is full of the golems and you can mine the crystal.
thanks