I don't think the "Most ratings" and "Popular" is really a good idea. I'm afraid it will give the most rated games even more ratings and make the most popular games even more popular when i think the most optimal outcome would be getting every game the same amount of ratings.
So I've considered this a bit, but I think having the "popular" and "most rated" sorts as options is better. Here's my thought process
- The number of ratings is not equal to your score, the person with the most ratings is not the winner. You can think of the number of ratings as a confidence level of how accurate the score is. If someone gets one rating with a score of 5, there's a very low confidence that it's correct. If someone gets 100 ratings averaging 4.5 then that's likely to be their "true" score
- itch.io uses a weighted rating system based on the number of ratings. Entries that have below the threshold number of ratings will have their average score proportionally decreased. This threshold is dynamically calculated based on the median of number of ratings across all rated projects
- Ratings are open to everyone, not just people who submitted. This means that your average rater doesn't have the same sense of duty to help rate things that may need ratings.
- A "Popular" sort lets someone who is new to the jam view submissions that have been vetted in some way by the community. It will make it easier for them to be engaged with playing and rating jam entries.
- If their only option is random, they may immediately give up and not rate anything because the things they see may be of questionable level of completion, buggy, broken, etc.
- When building a product (or a game), if you make it easy for a user to get engaged with your project, then they're more likely to continue using it. Conversely, making it difficult for them will likely lead to them giving up and "bouncing."
- In conclusion, I think that having the option available will increase average number of ratings given to each project.
- I don't think it's helpful to think of it as a zero-sum game where if someone rates a popular game they won't rate another less popular one. I think it's more likely that:
- if someone rates a popular game they enjoyed, they'll be will to go explore and rate other games
- if someone only sees things they aren't interested in they will rate nothing at all and bounce
- I don't think it's helpful to think of it as a zero-sum game where if someone rates a popular game they won't rate another less popular one. I think it's more likely that:
Sorry for the long response, tell me if you have any concerns about my thought process. Also keep in mind, "Random" is the default view, "Popular" is another option in a menu. If you don't want to be influenced by it then don't look at it.
Thanks for sharing your thoughts
That may be my opinion only, but what you described is precisely what we wanted to avoid: people (meaning everyone) will go to the submission list, sort it by "popular" (thinking that these games are probably the most enjoyable), and play/rate a certain amount of them (probably not 50). At least that's probably what I would do, from an outside point of view. So the "top" games will get even more visibility/ratings. To quote you: "if someone rates a popular game they enjoyed, they'll be will to go explore and rate other games". Except that the next game in line would be the second most popular.
The same use case with "random" order would at least be more fair to games that are not on the top 50 (even if that's not perfect, I agree)
I'm not saying that having these kind of sorting is bad, but I would have activate it only once the ratings are closed.
what you described is precisely what we wanted to avoid: people (meaning everyone) will go to the submission list, sort it by "popular"
That's not what I described at all. There's plenty of people using the random order to find games. The idea is that the popular gives an opportunity for those that might otherwise give up and rate nothing a chance to find a way to get involved with the jam. Hope that explains
The same use case with "random" order would at least be more fair to games that are not on the top 50 (even if that's not perfect, I agree)
There's also a sort for least rated. Keep in mind "Top 50" is represented by their score, not how many ratings they have. Nothing about a project's score is made available until the rating period is over.
I don't think it's a good idea to show least rated at the default sort because there are a lot of projects that are broken or difficult to run that will end up getting stuck at the top of the list. This will push even more people away from getting involved with the jam if they're constantly shown those kinds of projects.
Random sort by default is the most fair to distribute ratings. The other sort options are there to let those who are curious explore. I think this is the best approach to encourage the most ratings per project across all entries to the jam.
First of all, I really appreciate how much thought you've put in considering this feedback.
So it seems there is two issues (or consideration) at hand: How to make sure rating doesn't get too concentrated to the top 1% and how to make it easy for people to engage in the voting.
To be clear I don't think the current system is that problematic so we discuss mainly on how to improve it even further.
- How to make sure rating doesn't get too concentrated to the top 1%?
Let me try to break down this one even further. I think it all come down to how to have a fairer distribution of game played. If a game is played, ratings will follow, comments, and so on... For new jammers the rating might not be important (and sometimes discouraging) but having positive comments can encourage them to continue participating in jams. So actually it could lead to a multitude of solutions like a optional queue system that would prevent user to pick the nicest thumbnail, or pre-installing game if you have the app...
The fear is that the voting period is the only few days many games will have the chance to be played. After the result come out, most interest will be focused on which games were the most interesting to play. Only a tiny minority of user will start to explore the submissions. So the voting period is a crucial and exciting period to have your game played by strangers.
At first I agreed with you that it is not a zero sum game. Yes a user rating a popular game doesn't mean they will not rate something else. However after sleeping on it, I start to change my mind again. Time is limited and time spend on a popular game is less time that could be used on other games that would need reviews. I struggle with that myself, I have a massive list of game I would like to play. A lot of them are popular already and I feel really guilty to not be able to spend more time on more confidential games.
To be fair the popularity aspect was there before the Popular filter. I am lucky enough to have a game that got a lot of downloads (I think mainly because of the thumbnail) and I don't see in the analytics that it changed anything to have now the Popular filter.
- How to make it easy for people to engage in the voting?
I agree that having an easy entry point is essential to help people engaging with the system. The counterpoint to that is that the Popular filter is actually an option and as a result will be overlook by the vast majority of players especially the one not familiar enough with itch.io.
And if they start playing games with the popular filter they will probably to continue (especially for GMTK with so many entries), there is even fewer chances they would decide to explore games randomly. Not impossible of course but unlikely.
To be clear I am also of the opinion that only the participants should be allowed to rate games during the voting period. (someone who didn't participate can still rate and comment on the normal game page). I have also the impression that non-participant comment tend to be less encouraging because they don't have the same perspective than participant. So in general I would pay much less attention to these users. You have of course a totally different point of you since you need to consider this specific part of the site but the whole website experience.
In conclusion
After thinking a bit more about it, I would say that having the Popular and Most rated filter has probably very little effect. If there is an effect I still believe it would be more favorable toward already popular games. However it has an effect on the jam participants that already struggle to get their game played and seeing these options feels unfair. Not having the filter would feel much more balanced during the voting period as all title would have the same discoverability. (so in the end I think it's amore about feeling than actual user behavior)
Absolutely agree. It was already bad enough that the same 20 popular games show up all the time in the "check out more from GMTK Game Jam" Popup after downloading a game. If you get a head start in votes through good social media advertising, that will lead to even more votes and soon every gamejam team needs a marketing guy :P