Yes sadly that's a problem with the Godot engine :( I will see what I can do to improve it.
Its not because of that, its because of occlusion culling is not supported in Godot
