Skip to main content

Indie game storeFree gamesFun gamesHorror games
Game developmentAssetsComics
SalesBundles
Jobs
TagsGame Engines

Rating Queue Feedback & Issues

A topic by leafo created Jul 12, 2020 Views: 10,135 Replies: 112
Viewing posts 1 to 56
Admin (5 edits) (+13)

For GMTK Jam this year (2020) we’re we’re introducing a new system to make the distribution of ratings more fair, and to help block and detect cheaters.

I’ve created this thread so you can give feedback or report any bugs with the new system.

Here’s how it works:

  • If you are the submission uploader, then you can rate other entries freely and nothing changes for you.
  • Everyone else, including contributors, must use the rating queue.
  • The rating queue has 5 submission in it at any time. Everyone’s queue is different, and randomly selected.
  • You can only rate submissions that are currently in the queue. In order to access new submissions you must clear an item from your queue.
  • You can clear a submission in two ways: Rate the submission or Report the submission
  • Please only report submissions if they are invalid in some way. If you are unable to play a submission because it doesn’t work for your platform please do not report it
  • After you’ve rated 25 entries in total, you’ll be able to rate things without having to go through the queue.
Pinned ReplyAdmin (2 edits) (+1)

Update: You can now set your preferred platform on your queue page. Browser games will always be included, but you can set a preferred platform to exclude projects that aren’t available for your system. Hope that helps!

Update 2: You can now shuffle your queue once. If you’re not happy with the list of projects you see you can click shuffle to completely randomize your queue.

Also, some late submissions are still being processed. You may want to wait a bit before diving in to rate entries.

(+3)

When I click on the project page for the first game in my submission queue, I still can't rate it (and am told to check my rating queue)

Admin(+2)

Issue is now fixed, sorry about that!

(+3)

I'm confused because I'm at the game page of the 1/5 in my queue, but it won't let me review it.

Admin(+2)

The issue has been resolved and rating should be working as expected. Please try again, thanks

(+1)

Thanks! I like this system.

(+4)

Great game! Stunning visuals and challenging game play! Kudos!

Admin(+5)

This thread is not for leaving feedback on games. Thanks

(+6)

My queue is filled with Windows games but I am running Linux, making it impossible for me to clear my queue.

Admin (2 edits) (+2)

Thanks for the feedback, we’ll have a solution for this shortly, please check back later.

Edit: You can now set your preferred platform to filter out things you can’t play, and I’ve also added a shuffle button you can use once

(+1)

Same issue as @Kchortu and @gungancrab

Admin

Try again, thanks

(+1)

Can't rate any of the games in my queue. each of my queue entries just says go to the queue, you can't vote for this one. How can I clear the queue if nothing in the queue works?

Admin

Temporary glitch, but things should be working as expected now. Give it a shot. Thanks

(+2)

Two questions:
1. Can I filter my queue for browser only games? I have no intention of downloading & installing anything.
2. I've selected the one browser game in my queue and it won't let me rate it. Any idea what might cause this?

Admin (2 edits) (+1)
  1. Yes, going to enable this soon, so please check back shortly. Edit: this is now there
  2. Was a temporary glitch when opening first opened up, you should be able to vote on things now

Thanks

I cannot rate the games I'm asked to rate like many others here.

Admin

There was a temporary glitch when rating was first opened but the issue should be resolved now. Please try again. Thanks

I cannot rate the games, I already have rated 26 but I still cant...

Admin (2 edits)

Could you explain more about the issue you’re having? What do you see and what do you expect to see? Did you submit something yourself?

On the account you posted this comment with I do not see any ratings for any jams, or any submissions to any jams.

Thanks

yes!

I want to rate a game from a game jam but i receive the message "Once you have rated 25 submissions you are free to rate any submission." 

I have rated 26 but I still receive this message :((

Admin(+1)

I don’t see any ratings on your account. I have a feeling you’re using itch.io’s global ratings and review system, and not rating projects on the jam submission page. You must rate jam submissions on their submission page, as each jam has specific criteria you have to rate by.

Hope that helps

you are right!! I could make it right now!

The thing of need to submit a rate on a specific spot is a little bit confuse :( 

thank you so much!!!

(+1)

The jam host has told us that we’ll need to use the randomised queue if we didn’t “make a game” - does that mean be a game author, or the person who is hosting the game on their account? I need to use the randomised queue, but my partner doesn’t seem to - we’re not sure what’s going on!

Admin(+2)

I’m working out the details with him, in the meantime I encourage you to rate games. Rating random games only makes the jam better for everyone! Thanks

(+4)

Is it possible to make it so that you don't have to go through the ratings queue system if you're a collaborator on a submission but didn't submit it yourself?

(+5)

Is there a way to skip games in the queue? My queue is full of Windows-only titles, but I don't have a computer running Windows.

I am having the same issue.

Admin

I just pushed an update that will let you set your preferred platform. Hope that helps. You can find it on the page where you can access your queue.

I set my preferred platform to Linux and hard-refreshed to verify that the setting was saved, but my queue is still full of Windows titles.

Admin

If games also include a mode that can be played in the browser then they will also be included in your queue regardless of the platform you set. There are many games that are both for windows and the browser, since those were the required platforms to submit to the jam.

(+9)

I get why the queue is there but 25 titles is honestly so much that people who just wanna like rate their friends games will just give random or bad ratings.

I'd reduce the queue to like 5

Admin (1 edit) (+2)

The whole point of the system is to prevent people just signing up to rate their friends games, as it is unfair to those who do not have a lot of friends.

If people do create fake ratings to work around the queue, then we will know and disqualify all of their ratings.

(+3)

How?! So you rate ratings? How will you differenciate between someone doing random ratings vs someone rating it legit?

(+1)

hello, my friend cant view my link, it says that the page is not available

https://wargojam.itch.io/deal-with-your-s

Admin

You never published your project page, it’s still set to draft. Please try to avoid posting general support questions in this thread, as this thread is specifically for people who are having issues with the updated rating system. Thanks

Deleted 3 years ago
Admin(+1)

I added a preferred platform option that should let you filter out downloadable games. You can find it on your rating queue page. Hope that helps.

We’re also going to give everyone a free queue scramble shortly, so they can reset the games they’re getting if they happen to get stuck with things that they can’t play, but don’t deserve to be reported.

Thanks

(+1)

I only have mac, but my queue is filled with windows games! Is there a way to only get mac or browser games?

Same. This should be an option - I would like to rate games!

judging by other comments theyre working on/or maybe have a fix for adding browser games to queue, but the game jam said games had to have a windows or browser version to get in and didnt mention mac, so its likely most games dont have a mac version

Admin(+2)

There’s now a “preferred platform” option that you can find on your rating queue page. Try it out! Thanks

It's perfect, thanks for your fast response!!

(+2)

Could it be possible to see how many games we have left to rate before we can start playing games we want to play and rate those? half the ones i've had in my queue are unplayable or unfinished, i'd like to look for more complete ones to rate im just not sure how many ive already rated at this point

Admin

You can see how many submissions you’ve rated by going to the submissions page: https://itch.io/jam/gmtk-2020/entries

If you’ve rated any entries, then there’s an option to filter to projects you already rated.

Hope that helps!

I have a mac but lots of windows-only games in my queue, preventing me from rating anything. There should be a filter just like there is for the randomizer.

Admin

There’s now an option to set your preferred platform so you can avoid getting stuck with a queue of games you can’t rate. Try it out, you can find it on your rating queue page! Thanks

Thanks!

(+1)

Same issue as someone mentioned above- I'm on a Mac and my queue doesn't have any more browser/mac games in it. I'm mostly just finding random games from the twitter hashtag and playing them (not singling out friends games or anything), but I can't rate them which is annoying.

Admin

There’s now an option to set your preferred platform so you can avoid getting stuck with a queue of games you can’t rate. Try it out, you can find it on your rating queue page! Thanks

(+1)

please add a skip, I hate having to leave straight one star ratings because I can't get a game to work (ether downloading or controls)

Admin

I’ll be adding a shuffle button that you can use only once. Hope that will help you work around games you can’t play. I also recently added a “preferred platform” option so you can filter to just browser games.

I've rated and reviewed over 25 games in my queue, on my account page it says I haven't rated any!

Admin

I don’t see any ratings on your account. I have a feeling you’re using itch.io’s global ratings and review system, and not rating projects on the jam submission page. You must rate jam submissions on their submission page, as each jam has specific criteria you have to rate by.

Hope that helps

its get the theme but for me its impassible to control

(+1)

It would be very nice if we could rate game not in our play queue, with the understanding that the vote will not count yet. I stumble upon games randomly, and it is frustrating to not be able to make a note of what I thought of it at the time

Admin

Maybe an itch.io collection will work for you? Use the add to collection button to add the game to a list that you can check later. Hope that helps

(+2)

It does a little, like keeping the tab open, it's just that it doesn't feel as good to play other people's game on the discord or browsing randomly as I'm waiting to finish emptying my queue first.
I understand the need for a queue, and am not contesting that, just wanted to give feedback that it's like a barrier to playing the other games

Our lead submitted our game then tagged us as contributors yet we are still considered outside voters. Why?

Admin

We may end up changing this, but the rationale is that teams should not have more voting power than single participants. I encourage you to rate as many random games as possible because it makes the jam better for all involved!

I fully intend to rate as many games as I can but right now I can only rate games that show up in my queue. and every time I rate one then return back to my queue the queue has refreshed and now has all new games to look at. Many games I have already found and liked and wanted to rate I have been unable to because they are not in my queue. it is frustrating and not sure if this is intended or a bug. It does make me feel bad because I cannot rate the games, just comment on them.

Admin (2 edits)

The queue may shift if new games are added into the jam. I know that Mark is still processing some late submissions, but as long as you’ve opened the page to rate a game from your queue you will be able to rate it, even if it gets pushed off your queue.

Many games I have already found and liked and wanted to rate I have been unable to because they are not in my queue.

This is the point of the queue, to prevent people from only rating “good stuff”, or using their friend groups to boost their games. By forcing people to rate entries randomly at first we can get a much more fair distribution of ratings by the end of the jam. I feel that 25 ratings is a very reasonable number of minimum ratings for anyone to complete during the rating week.

Thanks for understanding

(+1)

Thank you for the clarification. I agree with this approach now after giving it some thoughts. Good luck with the feedback. Your doing a good job so far.

Is there any way to see how many games you've rated? It would be nice to know this.

Admin

You can see how many submissions you’ve rated by going to the submissions page: https://itch.io/jam/gmtk-2020/entries There’s a filter for the things you’ve rated

Great initiative but the games being linked are already highly rated

Admin

The games are randomly shuffled, and the number of ratings is not a factor.

hey, i'm one member of a fairly large (8 person) team for one of the gmtk submissions. our itch accounts are all properly credited in the submission but except for the account that actually did the submission none of us are able to bypass the ratings queue

(1 edit) (+3)

Would be nice if there was a counter of some sort to show how many of the 25 games we have cleared. 

It does feel a bit too much, I get that this system was introduced to make it fair for people who don't have many friends still get their games rated, and to prevent those with many friends getting biased ratings on their games. However I do feel 25 games is a bit much to play and download, as a public user. That takes up about 2-3 hours of my time (I think it's also because I like to leave comments on the games I rate). I think 5-15 games would be a more reasonable number. EDIT: It's been 2 hours since I started rating, and I've only rated 8 games, I really don't like this feeling of spending so much time just to be able to rate my friend's game or other people's games I want to rate freely.

I read some of the earlier comments and saw that a filter has been introduced to filter out games that need to be downloaded (for people who are unwilling to download games to rate them), but then this could be potentially unfair and skew more ratings to be given on browser games rather than downloadable games.

Just my two cents. :)

(1 edit) (+2)

I have some general feedback. At first glance the system looks fair and unique, but it has a major positive feedback loop flaw (somebody here is watching Mark Brown!): once games are being highly rated, they have a higher chance to be rated again and again - because the raters can choose from a list of 5 games and probably just choose the most popular ones on the list (that's what I did). On the other hand, this snowball effect is important to find the needle in the haystack as the rating progresses.

I think it would make MUCH more sense to hide the rating information and only publish it in parts, maybe something like that:

  • First 72 hours: don't show any data on the rating page, just give random games to raters.
  • Next 24 hours: show a boolean indicating if the game was rated (>=1) or not.
  • Next 24 hours: only show amount of ratings in gross scale (>5, >50 etc.)
  • Next 48 hours: reveal all the information.

Another thing to consider: if you ask people to rate 25 random games first, you might as well block them from rating any arbitrary games ever (not even after 25 ratings). That might incentivize people to randomly rate games in order to get to the 26th rating (there's no way to enforce the raters to actually play!). I wonder how many of the raters just rate everything but one game with 1 star.

On a personal note: we made what I think is an amazing game, and we get great feedback from our friends and colleagues - but we get so little feedback from jammers and raters and it's really a pity for us. We learned a lot and had a hell lot of fun, that's for sure - I just wish the rating distribution would make more game makers notice our game and give us the honest feedback we need so much.

Admin (2 edits)

I think this is great feedback. I think the issue of people always choosing the best game is unavoidable though, since even if the counts aren’t there, the viewer could go and look at the pages and select the best looking ones. There’s always going to be some bias since we let people rate from a queue size of 5. I’ll think about updating the queue page though for future jams.

I wonder how many of the raters just rate everything but one game with 1 star.

There’s no way I can think of to truly block people cheating or trying to work around the system, but at least something like this can make it become more obvious. At the end we can look over the results to identify raters to disqualify.

It’s the same idea for both issues, we can’t 100% remove these types of issues but we can have systems in place to try to mitigate them and get closer to something that is fair.

Personally 3 stars is 'average' to me, if you get 1 star then you did it really badly.

I have a Java Issue on my browser and my list is full of "play in browser" games, So I can no longer test other games.... Even if I clicked "Windows" game in my preference. Thanks...

Admin

The browser based games have nothing to do with Java and will work in any standard browser. No plugins necessary. Hope that helps

(2 edits) (+1)

So I think having to rate 25 games is a bit much, but I'm doing it anyways to be able rate some other ones i saw.
But what I find frustrtaing is, not being able to see how many games I've rated up till now on the rating queue page or something.
Is there anyway that can be added?

(+1)

Just finished my queue and I have to say: It was a pretty painful experience.
25 is just way too many games to reasonably expect people to go through.
It's honestly pretty sad, because you can't give the individual entries the time they deserve.
I actually think the queue is a good idea, it just shouldn't be quite as long - maybe 5.

(1 edit)

Feels to long. I'm ignoring the suggested games when I download one because I depend on the queue to be able to rate. It discourages exploring any other way because you only get limited participation. I avoid the main page because I've seen cool thumbnails but I don't want to click on them until I get the ability to actually rate those games. If I play a game any other way it's extra steps to ake sure it gets rated, and the rating isn't even fresh anymore.

That's disappointing, I wish I could just rate the games I want to rate, instead of being forced to play random games.

Hello! i was just wondering if there was a more elegant way to check my queue other then going to a game in the submissions tab and clicking the "view your queue" button?  

Admin

There’s also a direct link to your queue if you go the jam’s submissions page: https://itch.io/jam/gmtk-2020/entries

(1 edit)

I'm getting the "public viewers must rate via the queue" message despite the fact that I submitted a game. My partner made the itch page for our game and set me as a co-creator, do I need to do something else to my account?

Ah, I see the comment above. I'd like to lobby for this to be changed for next time. It doesn't feel great that we're equal partners but my account ended up with second-class-citizen voting status. Maybe cap it at like 4 voting-rights contributors if you're worried about voting power disparity? 

can't rate a game. 

Admin

Do you have any more information you can share? What to expect to see and where? I don’t have any reports of other people having trouble with rating.

Can I see how many I have rated so far

Admin

Yes, if you head to the entries page there’s a “Rated” filter that will show how many projects you’ve rated.

Hi, I have an issue with rating, I made a game on a team and I have admin power, and Im added as contributor, double checked. However I cant rate games until i dont rate 25 games, supposedly I should be free to rate. Any solution i could use? Thanks in advance! 


My game: https://itch.io/jam/gmtk-2020/rate/694677

Admin

Contributors can not freely vote on entries, only the original submitter of the team has that ability by default. This is done to prevent larger teams from having more voting power. Please read the original post for more information about how the system works. Thanks

Randomly playing and voting on entries is beneficial to everyone in the jam, as it makes it more fair for all involved.

i get the reason behind its creation but surely there is a better way to handle it. i dont want to have to slog through  voting on 25 random games i will never play just so i can rate the one game i did play. maybe you have some sort of timer thing, or only established accounts that meet specific criteria can vote

Admin

voting on 25 random games i will never play

If you aren’t playing the games then don’t rate them. The whole point of this system is to prevent people creating new accounts just to rate their friend’s submission.

It would be good if you can flag certain games that are broken / can't be played. If not, it ends up clogging the queue. If people flag randomly, that can be a checker for a spammer/fake account.

Admin

You can do that, please use the report tool. There’s a button on the submission page to report the game in question. Hope that helps!

Why do I have to fill out 2 captchas on the same page - one to rate and one to comment? Then one captcha expires while I comment and I don't know, if I have to rate the game again? Or comment again? 

What's a "Game Jam"

Admin (1 edit)

Please don’t post in old topics if you don’t have anything to contribute that is relevant to the topic.

A quick internet search can tell you all about what game jams are: https://itch.io/docs/creators/game-jams

anypone wanna join my jam

For the OST composing jam that I'm participating, so far I could only rate one submission and my queue is empty after that:
https://itch.io/jam/ost-composing-jam-crunchtime/entries

You can't rate this submission yet

This jam requires you to clear your rating queue before you can rate any submission you want. Please check your queue to see what submissions you can rate.

Admin (1 edit)

Due to the nature of submissions on the OST jam, please do not select a preferred platform, as OSTs typically are not executable and don’t have a platform, so you end up excluding yourself from untagged submissions. I’ve reset the preferred platform on your account for that jam.

Thanks

Thank you.

Interesting! i'm excited to see  how this works!

Admin moved this topic to Feature Discussion

HI! I can no longer see games to rate in my queue in the Shadowdark jam due to my inadvertently hitting the platform button. Please assist! Thanks.

Admin

You should be able to update your platform again from this page: https://itch.io/jam/shadowdark-independent-game-jam/unrated

Tell me if you are having issues

Still not working! I've rated 58, would love to rate the rest. Thanks!

Admin

Can you give me an example of a game that you think you should be able to rate but you can’t?

the Lantern Tower

Admin

We’ve changed it so you can now remove your platform preference. Please go to https://itch.io/jam/shadowdark-independent-game-jam/unrated and select the (reset) option under preferred platform.

Hi, as someone participating in my first game jam, I want to point out a problem with the preferred platform function. Since the game jam in question revolves around written material for a tabletop roleplaying game, the majority of the entries don't really fall under any platform due to simply being written documents and artwork. Thus, if someone curious tries to see what the preferred platforms option does, they basically lock themselves away from rating most of the entries.

Someone in the community posts ran into this problem and in an effort to help them I tried replicating it and sure enough, there seems to be no way to reset the preferences as far as I can tell.

Also I notice above there's mention of a 25 entry limit that could be reached to enable rating submissions without the queue. That doesn't seem to be the case.

Now, I realise this thread seems to refer to a specific game jam as opposed to the rating queue in general so I understand if this is the wrong place to voice these concerns. That said, I would appreciate if anyone could point me in the right direction.

Admin(+1)

Since the game jam in question revolves around written material for a tabletop roleplaying game

I can definitely see the issues this might cause in a jam that is primarily non-platform specific content.

We’ve gone ahead added the option to go back from a platform to no preferred platform set.

Also I notice above there’s mention of a 25 entry limit that could be reached to enable rating submissions without the queue

This is a setting of the jam that is set by the jam host

Fair enough. Thank you very much for your quick action! I'm glad I reached out and I admire your efforts

Hi! I tried to set up a rating queue for my jam here https://itch.io/jam/velox-fabula and I'm getting reports that no one can see a queue. I sent in a support ticket, but figured I'd also post here in case it helps this get resolved faster (I'm trying to avoid some games getting an unfair amount of votes versus others). Any help is greatly appreciated!

for my settings i have the following selected: 

Who can vote on entries? - Submitters  & contributors 

Rating queue - All voters 

Rating queue size - 5 

Rating queue unlock size - 5

Hello, I activated the rating queue on this jam: https://itch.io/jam/alternative-soundtrack-music-jam-5
Yet nobody was affected by the queue, as if it was not activated.
There are the settings: 


There may be a bug when the rating queue unlock size is superior to the number of entries?
Still, on an older jam I tried to leave a blank for the unlock size and it didn't worked either.

Have a good day.

Admin

Jam hosts are not subject to the rating review queue if I remember correctly, so the UI that you see will not be what regular participants will see. I’m not currently aware of any issues that prevent the review queue from working as expected, can you share why you think it wasn’t working?

Thanks

Several jammers reported to me they had access to every submission for the rating, they were unaffected by the rating queue.