Did you have conversation with the elf in witton? If you already did that , then:
Goto Calbridge Stronghold
Go behind the gambler ( a one block path on left back of gambler)
It will look like dead end but it's not & Spam space bar while you go up & down.
If it still doesn't work just do other quests .
I also had trouble , I wasn't able to trigger until I finished most of the quests (Endgame) . It may need you to finish specific quest in order to trigger open the entrance.