itch.io

Oh wow I posted this years ago but yeah the problem still stands. I entered a jam recently where the most popular entries were topping out at less than 30 votes (niche interest area running a friendly jam). I'm assuming in order for the lower half of the field not to get heavily penalised for having a few less votes than the top half, the people running it actually did their own manual calculations for the results release. It seemed unfair for an entry to slide a few places down in the rankings (and in one case affected who won a category) because of the difference of a few votes. Being a niche area means it is quite hard to get "lots of votes". More interaction can mean more votes which is a good thing, but when the difference can be in the single digits as to where the penalty line is, it's pretty disheartening for games that slide under that barrier. (and let's face it, about half the entries in a comp are going to go that way no matter how interactive everyone is or is not. My game was not one of those affected this time so that's not why I'm complaining, I just feel for some comp types it isn't very fair.)

In theory it's a good idea to stop an unpopular entry with a handful of perfect votes from the dev's friends winning over a more popular and better made one with 200 votes over a range of scores, but it just doesn't work for the smaller jams where if you get 20 votes your score might be unaffected, but if you get 19 you're penalised. An option like Igor suggested or to just work off raw scores rather than calculated ones would be good for those. I'm assuming not an option though considering it's been this way for at least the 3 years since I posted this thread.

As a submitter, the goal isn’t to avoid the “penalty”. The point of the “penalty” is to allow for a way to rank every game against every other game as fair as possible.

I think the mistake here is referring to it as a penalty, because it suggests to people they did something wrong with their submission. In reality, it’s more appropriate to call it an adjustment, as it’s serves as a mechanism to more fairly rank every game against every other game.

As an example, if your game got 10 votes with an average rating of 4.5, and someone else got 100 votes with an average rating of 4.5. Do you think it’s fair they rank above you? I think so, since they’ve managed to collect that many more votes while still maintaining a relatively high score.

If there is a fixed number minimum number of ratings required, then it’s going to create a distortion in rankings to the people who collect exactly that number of ratings. A lower number of ratings creates higher variance in the average score compared to a submission with a higher number of ratings. There is a chance the lower-number-of-ratings project has a non-representative score that could bump them above a game that got substantially more ratings.

you get a few ratings and think it’s enough

If you care about your results in a ranked jam, then I recommend participating in the voting process throughout the duration. Rate other people’s work and leave comments. It’s the best way allow others to discover your submission. If you’ve only collected a “few” ratings, then it’s unlikely your raw average score is representative in a way that can be compared to submissions that received substantially more ratings.

how many ratings are needed at least to not get the penalty

The goal is not to tell you to hit a target number of ratings, as that essentially encourages people to try to game the ranking system. Instead, your goal as a participant is to continue to collect ratings. The more ratings you collect, the more “accurate” your average score is relative the number of ratings collected by all other submissions.