Just got Internet.. and yes you were correct. I had forgotten that the requirement was to leave all the way to the world map, not just the building.
