Have you accepted his invitation? He asks you after you finish "Jungle Jumping", after you get back to camp (and have a small dispute with the other explorers), for example. You can't just walk into the building, you have to accept his invite and be invited inside.
If he's not letting you in, and you can confirm you've accepted his invitation, then we'll look into it as a potential bug.