I found a previous thread asking this same question, but with no response. That user shares my thoughts exactly on the issue, but to reitertate: in a ranked game jam, each game receives two scores: Score* and Raw Score. Raw Score denotes the actual ratings a game received from its players, while Score* seems to be one of two things:

• If the game was rated by at least the median number of players (regardless of how well or poorly it was rated), Score* is the same as Raw Score.
• If the game was rated by a below-average number of players (regardless of how well or poorly it was rated), Score* is lower than Raw Score. The fewer players a game received, the more it is penalized.

Entries are then ranked by Score* instead of Raw Score. I am strongly against such a scoring system, because:

• It gives games a scoring advantage if they have an eye-catching thumbnail or if the developer has a preexisting following.
• It gives games a scoring disadvantage if they belong to a niche genre.
• Having two separate scores with no explanation for the difference is confusing to developers.
• Games with a high Raw Score can have a low Score*, making developers feel like their game was much worse than it actually was. To use an extreme example, this Trijam game received an overall Raw Score of 3.833 - not bad at all! - but since it was only rated by 3 people, its overall Score* was brought down to 2.213 - a pretty bad score. This had a huge effect on its ranking, as well: it would've ranked 12th out of 35 by Raw Score, but it actually ranked 28th out of 35 by Score*.

I have a guess to why itch does it this way. When a game has very few ratings, there's a greater chance of the rating being skewed up or down by a particularly opinionated player, or in less scrupulous circumstances, by a willing accomplice/salty troll attempting to artificially inflate/deflate a game's ratings. I can see the logic behind behind awarding the victory to a game that got, say, an average of 4.5 stars from 20 players, rather than naively awarding it to a game that has 5 stars because only one person played it and that one person happened to give it 5 stars everywhere.

However, this scoring system seems like a poor solution to that problem. Yes, a game's rating may be biased if only a handful of people have played it, but there is no way to tell in which direction the rating is biased. Fewer ratings are not necessarily a reflection of lower quality; they are a reflection of higher uncertainty about the game's quality. But this scoring system doesn't reflect that uncertainty- it simply tells developers that their games were worse because they got fewer players. Also, the median number of ratings is much too high a threshold at which to start punishing games for having too few players- you're effectively disadvantaging the entire bottom half of the submission field sorted by popularity. I'd think half the median, or better yet, a fixed number of ratings like 3 or 5, would be a better threshold to determine if a game has too few ratings to rank well. Above 5 ratings, I'd have a hard time claiming that any one player skewed the numbers too much.

To me, a more honest system would be simply not ranking any game that doesn't receive enough ratings. It still prevents the 5-star scenario I described above, but it doesn't tell the developers that their games were worse than the more popular games- just that not enough people played their games to get an accurate assessment of how good the game is.

If not that, I'd at least like the option to turn off the Score* system- if I were running a ranked game jam, that's the first thing I'd do.