Wow, I feel all of this. Keeping your motivation could be a topic all on its own, to be honest. I am glad you chose to enter even though you hit that wall! Keeping motivation - especially when you've overscoped - can be a serious challenge. I absolutely get it!
As for that moment when you realize the game 'isn't fun'... that is a tough pill to swallow. There are plenty of times where I've had to go back to the drawing board for that exact issue. During a game jam, that is even harder since we're under such a time crunch. The prototype phase can be critical in terms of that search for the fun.
I hope you will consider continuing development - in this project or others - once you have recovered that motivation. The inspiration to create comes in bursts, sometimes. I also completely agree with the idea of knowing when to walk away from something. Seize it when it is there, but don't force it. You will get there - and you won't resent having done so! :)