Man how time flies. I looked into it a bit more and it looks like I can just write an "AssetBundle Loader" so that modders can just export AssetBundles from Unity and players can import them via a mods menu/Steam Workshop support. I'll try and get on that soon ^^
It'd take some additional work, so I'll avoid it if the AssetBundle approach renders it unnecessary. 3D model importing for example can be done as an AssetBundle but it is a little less convenient since the modder must download Unity first. We'll have to cross that bridge when we get there.