What happened was that I grabbed the two gems up there and touched the door, but I was rewinding while that happened, it put me under the wall. Maybe a box collider (trigger) underneath to disable the collision until the player is out of the area, that way you can always pass through it from one side but not the other?