It was recommended that I post here with concerns that have been discussed in the discord server. First things first, I want to say that I hate complaining, and wish that there was a better alternative. I would also like to make sure that what I am saying is not misinterpreted- I think the vast majority of the games that are ranked highly deserve to be there and to be judged. Finally, no matter what is decided, it will leave someone unhappy, barring the judges ranking every single entry, which of course is infeasible. I am not prescribing what to do- I honestly don't know what the best option to do at this point is, but I do feel the need to bring the issues discussed here. I will offer my thoughts as to how to potentially avoid some of these issues in future jams, but obviously you can take or leave those.
I have talked with other members in the discord channel and it's unfortunate, but it seems like there are some pretty severe issues with the ranking system. One of the worst problems that seems to be the case is that many of the games that were near the top of popularity (which I assume is based on how many people played it and/or how much time was spent playing it) received strategic 1 star reviews. Obviously it's impossible to prove that these are malicious, but it seems incredibly unlikely. To be clear, this is a problem that our game suffered from, but it was not only ours, and seemed to be worse for games that appeared to be doing well. I think it's saying something that our game was consistently near the top of popularity (most often within the top 3), yet fell into about the midpoint of the ranking system.
At the end of the day, I really don't know what the best option to do is. If you choose to have judges rate 20 games total, and choose the top 15 ranked games and 5 other games, the people ranked 16-20 will feel crappy. Although I suppose that it would suck for the people in 21st place as well, regardless. Hopefully this can at the very least start a discussion to find a good solution.
I posted this in jest in the discord server amidst the discussion, but it highlights many of the problems with an open vote system, and is the reason most jams used closed voting. Here is the abuser's guide to winning open voting jams:
- Do the absolute minimum to promote your game. Make the title and cover art as generic and/or poor as possible as to lower the likelihood of people playing the game
- Near the end of the voting, have friends and/or bots review your game
- Strategically rate competitors 1 star, and games you deem as manageable competition higher. If you're using bots or friends to do this, it's untraceable anyways
It's impossible to say if anyone cheesed the system, and I don't want people to think that I'm making that claim. I do definitely believe that there were people that actively did the last part based on people I've talked with, however.
Finally, for future jams I'd recommend two main things for jams in the future:
1. Please, please, please use a closed voting system. There are obviously a few downsides to this (namely the number of people that can rate games), but it makes abusing the system much more difficult. It incentives participants to actually try to get other participants to play their games and vice versa. While you can still 1 star games, it is traceable so if there's obvious strategic voting going on at least people can be called out on it.
2. This is more minor, but having more than 1 voting category is often useful to get a more creative variety of games and give smaller/solo teams a better chance. Keep the overall rating as it's own category, and use that as the primary info for voting, but add categories for other aspects (e.g. "Art", "Audio", "Gameplay", "Theme interpretation"). This way you can get a larger breadth for finalists- games that might be really good in one or two regards but didn't have time for the rest. This second point comes more down to what you're looking for, but I'd argue that it would incentivize participants to be able to focus more on aspects that they might be able to compete on.
tldr: It seems apparent that there are some pretty major issues with the voting and that it should be discussed how to handle it