Number of ratings should be factored in, not excluded, somehow. It actually reflects the community's view of a game. Right now it feels like games with high number of ratings are the ones being penalized, because they're more "exposed" to bad ratings. Many games with less than 20 ratings are topping the Design list.
Viewing post in JSON export of jam results + results discussion
For example: Gooey Castle ranked 1st with 73 ratings. Totally awesome. 2nd place? 11 ratings. Doesn't feel with the same weight at all.
My definition of cheating is when someone creates fake accounts to create fraudulent votes on a submission.
I manually went through all of their ratings, and I don't see any evidence of that. It's very likely that they have access to a lot of distinct people (their friends) who were willing to go and create itch.io accounts and vote on their entry.
That's not cheating, but a disadvantage of public voting system. Many other submissions did the same exact thing, they just were the ones who happened to have the most friends.
I'm not accusing Gooey Castle of being fraudulent. That's another topic. All I'm saying is that there are games on the top 100 overall that may be great games, but weren't exposed enough to the community. On the other hand, we've got games with both high ranking AND high number of ratings; those are totally fine in my book (like TowerBag). Should every highly rated game be ranked higher? Not exactly. A popular game may be horrendous (#5 most rated for example) and should tank accordingly.
I'm not saying I've got the answer for this issue. I'm only pointing out that highly rated games are exposed to a wider perspective from the community, and that should have some value.
12 games from the top 100 most rated games made it the top 100 overall list.
A more positive example: Negative Nancy, #17 overall which is excellent. However, with 81 ratings, it should totally be among the top 3.
I guess the "impact" of having very few ratings should indeed be stronger. There are games in the top 100 that have only 8 or 9 ratings (all done by a bunch of friends, as comments are not written in english)
If the best stategy to be on top of the list is to show the game only to a few people (to ensure 5 stars), that goes against the idea of a game jam to me.
Some people know how to promote their game in the community forums and get dozens of ratings and comments, which has nothing to do with game design... As long as a game has more ratings than the median, it feels right to me to consider their score valid.
To me, voting shouldn't be public.
There could even be an option for a dev to be presented a random game to rate, and that rating could have more weight than non random ones, that way you ensure fair rating.
Yeah, for this jam we had a variant of the Randomiser to show us a random jam entry (which worked very nicely), so it could be interesting to see an option for only the Randomiser to be available when picking games (plus the top 100 least rated games list). Not sure how feasible, but interesting nonetheless.