Ok done a little research, I can use this tool in the meantime to optimise my files: https://github.com/donmccurdy/glTF-Transform/blob/master/docs/SCRIPTING.md
This reduces my file from 2.5mb to about 500kb, and that size will not grow much unless I add more unique meshes 🎉