After speaking to him multiple times, the elf guard should let you in at some point.
But that will happen after you have exhausted all the quests the Bartender says.
If you still cannot proceed, I will need to investigate the issue, as it may be a bug. Contact me at our main website at https://townofmagic.com/contact/ with a valid email so I can contact you.