I felt the exact same after I chose Harold for my first route. I legitimately didn't play for a month or two after that because I just loved it so much and felt so sad the time was up. I was hesitant of playing the other routes but I don't regret choosing to do so.
While it hurts to see the other guys whose stories you have seen be sidelined, it's worth it for the great bits you get with every guy. Darius is still probably my least fave, but the difference went from noticeable to barely any at all, and Spence easily became nearly tied with Harold for me.
Definitely not saying that you sticking to Spence only is wrong, he's incredible and I can't blame you for not wanting to go through other routes, but I promise you that you will find lots to love if you do, especially with Spencer in certain routes.