The only thing I've ever noticed is on your own personal jam dashboard... when you click on "joined jams" it lists ones that you didn't submit anything to, which is probably only annoying to me. I knew about the "Leave Jam" link, I just wasn't able to get to it in time.
Nobody will think any less of you. I've been developing games since the Commodore 64 was hotter than a two dollar pistol, and I haven't completed, like, SIX jams.