Man how time flies. I looked into it a bit more and it looks like I can just write an "AssetBundle Loader" so that modders can just export AssetBundles from Unity and players can import them via a mods menu/Steam Workshop support. I'll try and get on that soon ^^