Ah, I found it in the room with the cave crawlers. You were right about that Perception requirement, I had been in that space before and didn't find the entrance to the mine.Good thing troglodytes are easy to seduce and give a good amount of XP.
Also easy perception boost is potions and gear so you don't need to level up much or farm bosses for exp.
