The library file for unity is over 30k, it's taking way too long for me. Do we need to have it?

No, the library and temp folder are temporary files that you don't need. The best way to know what files you need is by checking the unity gitignore