To tell you the truth, I was horrified to play other jam games, as I thought we had no idea what we were doing and having smaller teams means worse games, but after playing them it was a wake up call for me - some people don't know what they're doing either. I guess the issue is not with team sizes per se as it is with project management. While compared to my friend I didn't do as much work, I made sure the scope was low and we had a clear goal to shoot for. It's much easier to make a game that you believe you can make with little unknowns.
If there's one piece of advice I can give is to team up with people that are enthusiastic to work with you, share your vision or at least that you have great chemistry with. It makes the toughest challenges much easier to deal with and the experience becomes that much more meaningful. If you lose, it's spread out evenly, but if you win, you all bask in the glory. I'm grateful I had a friend that could dig me out of a dark place at the end of day 1 and I probably would've bailed from the competition half-way. And he enjoyed working with me still, so that's always nice. I swear there was a whole character arc happening during those two and a half days, but I don't want to delve into the details here.
Thank you for your inspiring comment!