Interesting; I found the edge of the map (which is not a clean edge but there is no ground which was enough to fly through... oops) and discovered that you have ground collision under the map too... probably a waste of effort checking for that... I also noted that the building are rendered "up" towards me except that I'm underground so... don't know how that happened but definitely interesting! :)
Given the above I guess that your engine is rendered everything towards the player? Except that when they are underground there is no ground rendered (even if it has a collision plane - pun intended!) and, I guess due to that, rendering the scene takes way more effort as I guess the entire skybox has to be rendered, rather than removing the part below the ground (because it's not rendered it can be seen as so I guess isn't culled?)
Anyway, hope all goes well with you and good fortune on the dev work - whatever you're working on now :)