Maybe it's an issue with CPU, and it's that streaming taking resources from it, while also the game using the CPU to render materials as I didn't enable GPU instancing in Unity. Since it's not a WebGL game, it shouldn't cause any issues and I should probably enable those.
