GMTK Game Jam 2019

Hosted by Game Maker's Toolkit · #gmtkjam

2,552

Entries

46.6k

Ratings

Overview Submissions Results

Community509

Screenshots Submission feed

GMTK Game Jam 2019 community

[Update: added karma sort] Updated submissions page, report broken games, submission randomizer, comment linkbacks

A topic by leafo created Aug 07, 2019 Views: 3,617 Replies: 61

Viewing posts 1 to 22

leafoAdmin5 years ago (4 edits) (+13)

Hey all, I'm an itch.io admin and responsible for coding everything to do with jams. Firstly, I'd like to apologize for the slowness issues during the submission process last Sunday. It took me a lot of thinking but I eventually identified that all the slowdown was due to a single page, the submissions page. (It had nothing to do with the number of games being submitted, rather everyone trying to view the submissions as they came in). This is all fixed now and we're ready to handle any jam of any size!

I've been making a few enhancements to itch.io for GMTK:

New submissions page

Sine I was spending bunch of time optimizing the submissions page, and I figured I might as well rewrite the UI as well. If you're logged into your account you'll see the new (fast loading) submissions page with a lot more sorting options. Keep in mind that this page caches data about submissions for a few minutes, so wait a few minutes for rating counts to update.

If you have any feature requests for this page please leave a comment. Now that the code is fresh in my mind I'd be happy to consider any ideas to make it better for sorting through the large amount of games.

Updated "In need of ratings page"

If you didn't know, there's a page that lists 100 projects in need of ratings: https://itch.io/jam/gmtk-2019/unrated

There was a bug with the query that caused the list to be empty due to a large number of empty submission pages. The bug was fixed so you can continue to browse the 100 least rated games.

Disqualifying empty & broken pages

I spoke with our host, Mark Brown, and he agreed that broken and empty submission pages should be disqualified. Disqualifying entries will help clean up the submission list so people can focus on rating projects that actually work. To help enable this, I've added a Report submission button on all submission pages. Please use this to report anything you think be disqualified or removed. We'll going through the reports one at a time and taking any action if necessary.

Thanks to everyone who participated, you were part of the largest jam on itch.io to date!

leafoPinned ReplyAdmin5 years ago (1 edit) (+1)

I've started to disqualify broken or empty submissions. Please keep reporting things as you come across them so we can clean up the submission list for others so they don't waste time trying to play and rate things that don't work! Thanks

leafoPinned ReplyAdmin5 years ago (1 edit)

I've added Game Jam support to our Randomizer! It's a separate view that lets you go through projects pages one at a time in random order, skipping things you've already seen. Here's a direct link: https://itch.io/randomizer?jam_id=10205

I've also added a link to it on the top of the submissions list. Enjoy!

I also update the dialog that appears after you download a file to show a random assortment of games from the jam to make it easier to find new things to rate.

leafoPinned ReplyAdmin5 years ago (3 edits) (+1)

Another update as we get closer to the end of the rating period. If you leave a comment on someone's entry page, then a link and image are automatically placed below you comment that goes back to your submission:

I think commenting and rating those who take the time to comment is an excellent way to reward those who are putting effort into rating games. This should make it really easy to find their submission. (The gray submitted text also links back, but not many people know)

Because people are likely to go and check out your game if you leave a good comment, you can "grind" for more ratings now by commenting on people's projects. If you're struggling to get ratings please go and leave some constructive comments. Don't spam though, your posts will be deleted.

Lastly, I added a warning if you try to leave the page without saving your rating if you've clicked on any of the stars.

Enjoy

leafoPinned ReplyAdmin5 years ago(+1)

One last update before the rating period is over. I added a Karma sort. This one is inspired by Ludum Dare, where it's sorted by ratings_given - ratings_recieved. What this means is that people who have rated a lot, but not received many ratings are bubbled to the top. I'm actually going to tweak this formula a bit to give precedence to those who need ratings more, but I figured I should get it out quick since many people were asking for it. Enjoy.

As a reminder, after the rating period is over, the final results will be delayed for about an hour as we process the votes. Thanks

leafoAdmin5 years ago (3 edits)

Experimenting with a different formula, slightly different ordering but should give preference to those who are more in need of ratings, and prevent super-raters from getting too much karma (diminishing returns).

Formula is:

karma = ln(1 + ratings_given) - log5(1 + ratings_received)

If you have ideas for other formulas tell me and I'll try them out.

Edit: I also personally think that a karma system like this is flawed. I wrote about why, and my proposed solution here: https://itch.io/t/527869/karma-system-for-the-game-jams#post-895699

I added the karma sorting option as an experiment, something to try out and see how it works.

Yahia ZakariaSubmitted5 years ago

How about making a weighted karma system? The new Karma system looks only at how many games a participant rated so they can rate all the popular games to gain karma and ignore ones that actually needs ratings. I think we can weight the karma so that a rater gains more points if the game they rated had a low number of rates at the time of rating. Therefore, the participant will be incentivized to not only rate others but also rate the games that need rates.

Johnny DalviSubmitted5 years ago

I think that this is a good system, just need some tweaking over time but it has been proven to work on LD.
Perhaps a good idea to prevent this kind of abuse is to give karma just to whom comments on the game, giving feedback (and only once per game for each user). Thanks for adding it btw

NaskeSubmitted5 years ago(+10)

I don't think the "Most ratings" and "Popular" is really a good idea. I'm afraid it will give the most rated games even more ratings and make the most popular games even more popular when i think the most optimal outcome would be getting every game the same amount of ratings.

leafoAdmin5 years ago(+4)

Changing the default sorting order was a mistake! The default sorting order is now random, like it used to be.

Nic MagnierSubmitted5 years ago(+5)

I agree. And I don't think it's only about being the default option. Just having the option give people the opportunity to easily access games that get a lot of rating.

It would be best for these options to appear only after the end of the voting period.

leafoAdmin5 years ago (6 edits) (+3)

So I've considered this a bit, but I think having the "popular" and "most rated" sorts as options is better. Here's my thought process

The number of ratings is not equal to your score, the person with the most ratings is not the winner. You can think of the number of ratings as a confidence level of how accurate the score is. If someone gets one rating with a score of 5, there's a very low confidence that it's correct. If someone gets 100 ratings averaging 4.5 then that's likely to be their "true" score
- itch.io uses a weighted rating system based on the number of ratings. Entries that have below the threshold number of ratings will have their average score proportionally decreased. This threshold is dynamically calculated based on the median of number of ratings across all rated projects
Ratings are open to everyone, not just people who submitted. This means that your average rater doesn't have the same sense of duty to help rate things that may need ratings.
A "Popular" sort lets someone who is new to the jam view submissions that have been vetted in some way by the community. It will make it easier for them to be engaged with playing and rating jam entries.
- If their only option is random, they may immediately give up and not rate anything because the things they see may be of questionable level of completion, buggy, broken, etc.
- When building a product (or a game), if you make it easy for a user to get engaged with your project, then they're more likely to continue using it. Conversely, making it difficult for them will likely lead to them giving up and "bouncing."
In conclusion, I think that having the option available will increase average number of ratings given to each project.
- I don't think it's helpful to think of it as a zero-sum game where if someone rates a popular game they won't rate another less popular one. I think it's more likely that:
  - if someone rates a popular game they enjoyed, they'll be will to go explore and rate other games
  - if someone only sees things they aren't interested in they will rate nothing at all and bounce

Sorry for the long response, tell me if you have any concerns about my thought process. Also keep in mind, "Random" is the default view, "Popular" is another option in a menu. If you don't want to be influenced by it then don't look at it.

minimaulSubmitted5 years ago

Thanks for sharing your thoughts

That may be my opinion only, but what you described is precisely what we wanted to avoid: people (meaning everyone) will go to the submission list, sort it by "popular" (thinking that these games are probably the most enjoyable), and play/rate a certain amount of them (probably not 50). At least that's probably what I would do, from an outside point of view. So the "top" games will get even more visibility/ratings. To quote you: "if someone rates a popular game they enjoyed, they'll be will to go explore and rate other games". Except that the next game in line would be the second most popular.

The same use case with "random" order would at least be more fair to games that are not on the top 50 (even if that's not perfect, I agree)

I'm not saying that having these kind of sorting is bad, but I would have activate it only once the ratings are closed.

leafoAdmin5 years ago

what you described is precisely what we wanted to avoid: people (meaning everyone) will go to the submission list, sort it by "popular"

That's not what I described at all. There's plenty of people using the random order to find games. The idea is that the popular gives an opportunity for those that might otherwise give up and rate nothing a chance to find a way to get involved with the jam. Hope that explains

The same use case with "random" order would at least be more fair to games that are not on the top 50 (even if that's not perfect, I agree)

There's also a sort for least rated. Keep in mind "Top 50" is represented by their score, not how many ratings they have. Nothing about a project's score is made available until the rating period is over.

I don't think it's a good idea to show least rated at the default sort because there are a lot of projects that are broken or difficult to run that will end up getting stuck at the top of the list. This will push even more people away from getting involved with the jam if they're constantly shown those kinds of projects.

Random sort by default is the most fair to distribute ratings. The other sort options are there to let those who are curious explore. I think this is the best approach to encourage the most ratings per project across all entries to the jam.

Nic MagnierSubmitted5 years ago

First of all, I really appreciate how much thought you've put in considering this feedback.

So it seems there is two issues (or consideration) at hand: How to make sure rating doesn't get too concentrated to the top 1% and how to make it easy for people to engage in the voting.

To be clear I don't think the current system is that problematic so we discuss mainly on how to improve it even further.

How to make sure rating doesn't get too concentrated to the top 1%?

Let me try to break down this one even further. I think it all come down to how to have a fairer distribution of game played. If a game is played, ratings will follow, comments, and so on... For new jammers the rating might not be important (and sometimes discouraging) but having positive comments can encourage them to continue participating in jams. So actually it could lead to a multitude of solutions like a optional queue system that would prevent user to pick the nicest thumbnail, or pre-installing game if you have the app...

The fear is that the voting period is the only few days many games will have the chance to be played. After the result come out, most interest will be focused on which games were the most interesting to play. Only a tiny minority of user will start to explore the submissions. So the voting period is a crucial and exciting period to have your game played by strangers.

At first I agreed with you that it is not a zero sum game. Yes a user rating a popular game doesn't mean they will not rate something else. However after sleeping on it, I start to change my mind again. Time is limited and time spend on a popular game is less time that could be used on other games that would need reviews. I struggle with that myself, I have a massive list of game I would like to play. A lot of them are popular already and I feel really guilty to not be able to spend more time on more confidential games.

To be fair the popularity aspect was there before the Popular filter. I am lucky enough to have a game that got a lot of downloads (I think mainly because of the thumbnail) and I don't see in the analytics that it changed anything to have now the Popular filter.

How to make it easy for people to engage in the voting?

I agree that having an easy entry point is essential to help people engaging with the system. The counterpoint to that is that the Popular filter is actually an option and as a result will be overlook by the vast majority of players especially the one not familiar enough with itch.io.

And if they start playing games with the popular filter they will probably to continue (especially for GMTK with so many entries), there is even fewer chances they would decide to explore games randomly. Not impossible of course but unlikely.

To be clear I am also of the opinion that only the participants should be allowed to rate games during the voting period. (someone who didn't participate can still rate and comment on the normal game page). I have also the impression that non-participant comment tend to be less encouraging because they don't have the same perspective than participant. So in general I would pay much less attention to these users. You have of course a totally different point of you since you need to consider this specific part of the site but the whole website experience.

In conclusion

After thinking a bit more about it, I would say that having the Popular and Most rated filter has probably very little effect. If there is an effect I still believe it would be more favorable toward already popular games. However it has an effect on the jam participants that already struggle to get their game played and seeing these options feels unfair. Not having the filter would feel much more balanced during the voting period as all title would have the same discoverability. (so in the end I think it's amore about feeling than actual user behavior)

SteaneSubmitted5 years ago

Its pretty obvious that anything listed by popularity/higher ratings will garner more views. Youtube, steam, google play work the same way. The more it is featured at the top of a page, the more clicks it will get. When you use google to search for something, do you goto page 3? No you don't.

Tricky_Fat_CatSubmitted5 years ago(+1)

Completely agree , It's much more interesting to discover new games without knowledge of popularity before the voiting period ends.

mindcopgamesSubmitted5 years ago(+1)

Absolutely agree. It was already bad enough that the same 20 popular games show up all the time in the "check out more from GMTK Game Jam" Popup after downloading a game. If you get a head start in votes through good social media advertising, that will lead to even more votes and soon every gamejam team needs a marketing guy :P

leafoAdmin5 years ago(+1)

I completely forgot about this screen. I'm updating it now to show wider range of games. Thanks for pointing that out.

leafoAdmin5 years ago(+1)

This is fixed now

minimaulSubmitted5 years ago(+1)

Totally agree too.. Isn't it possible to deactivate these filters?

weslitoSubmitted5 years ago(+2)

completely agree.

This introduce a strong bias in the voting/rating process.

kotobSubmitted5 years ago

+1 for this

HarrySubmitted5 years ago(+3)

Here's an idea: a random game button could help people choose a game to play, as there are a lot of submissions and it could be hard to choose which one to go for.

HarrySubmitted5 years ago(+3)

Also wanted to say - kudos to Itch for handling the slowdown situation and thanks for the postmortems on Twitter and on here, really interesting to see how these things work :)

leafoAdmin5 years ago

Maybe a special version of our randomizer https://itch.io/randomizer that only pulls from a specific jam?

HarrySubmitted5 years ago

Yeah, that way you could probably filter by platform and genre as well!

leafoAdmin5 years ago(+1)

This now exists: https://itch.io/randomizer?jam_id=10205

HarrySubmitted5 years ago

Thank you!!!

WapitiSubmitted5 years ago

mark brown used an external number generator and add to switch between the random number generator and itch.io during his livestream

KoverasSubmitted5 years ago

I'd really like to be able to filter the games by genre... Before the AI overhaul, the genre (if any) was listed under the game's title, so I could just browser-search for it on the page, but now I have no way of filtering out all the puzzles and the platformers I have no interest in. :-(

Matt GlanvilleSubmitted5 years ago(+6)

Thanks so much for continuing to update the site and making actual useful, requested changes!

Just a small note but on mobile (chrome) the filters get squashed down so much that you can't read the text. They look OK in landscape, but not portrait.

I also think that it's too easy to forget to click 'Save' on rating submissions. I forgot a few times myself, then watched Mark's stream where he forgot a few times too. Maybe scores could save automatically as soon as you click them, or make the Save button more obvious, or warn you if you're about to leave the page with unsaved changes.

Anyway these are relatively small quibbles. Keep up the awesome work!

leafoAdmin5 years ago(+1)

Thanks for the feedback. I've fixed the mobile display issue.

Regarding forgetting to save ratings, I'll try to add an unsaved warning in soon.

Matt GlanvilleSubmitted5 years ago

Awesome, thanks! :)

feofentovSubmitted5 years ago(+1)

New filters are very cool! And reporting broken games seems an awesome idea.

If you don't mind, I'd like to share some thoughts on submission process regarding UI. The only thing frustrated me was that our game didn't show up in the list of my existing games on itch.io for submission. I thought it was due to server issues, tried to refresh page and other stuff, but it didn't work. It took me a while when I undestand that pressing "Add new project" button automatically submit game to jam and our game is already submitted. So it would be perfect to show already submitted game in the list of existing games for submission (if it's not a bug of other kind).

Thank you for your work!

leafoAdmin5 years ago(+1)

Thanks for the feedback. The UI for the dialog is something I plan to replace pretty soon. Adding a list of things you've already submitted along will definitely go there.

Mike RenSubmitted5 years ago

Hi Leafo! Thanks for the new updates to the submissions page. The filters help a lot to tell which games will end up on Mark's top 100. I have a quick question though, how does the popular filter work? Is it based on view/download count on the page? Thank you!

leafoAdmin5 years ago

The "popular" sort sorts entries by their global itch.io popularity score, which is a secret algorithm used to sort games on pages like https://itch.io/games

It''s not affected by jam votes or scores. It gives people a quick way to see what the community at large is interested in or talking about.

Mike RenSubmitted5 years ago

Ah gotcha. Thanks for the reply!

Nic MagnierSubmitted5 years ago(+3)

I still really like the ludum dare system where people how rate a lot of game tenmd to buble up at the top of the list, encourage everyone to rate. It's quite common for people to wonder how to make their game more visible and with this system the obvious answer would be "go and rate games". The main draw back is that it might encourage some people to randomly rate games without relly playing them.

So I'd love to have the default filter being semi random, with game entries having a weight based on the ratio rate_given by rate_received

Would be also nice to have a text filter. A simple text input and when you type it filtered out games that doesn't contain the string in the properties of the entry (name, author, platform etc.) I can do that already with the search field in the brwser but somehow I would find it more convenient as a filter option.

Kitty FactorySubmitted5 years ago(-1)

Entirely agree with you about the Ludum Dare system. Even if there are some excess sometimes, it works very well and encourage the jammers to vote and comment in order to gain visibility. There is also a bonus in the scoring system when the creators give a thumb to the comment, so being constructive is really helpful.

I'm in favour of your semi-random filter too! :D

leafoAdmin5 years ago(+2)

This is something we used to have but was lost in the shuffle a while back. I haven't thought about how to handle team situations yet. Since a project with 5 members may have 5x as many ratings than one with 1 member. I think maybe taking the max of each member is the most fair? In any case, I do want to add this back before the rating period is over.

Sagewind StudioSubmitted5 years ago(+5)

I know this is kind of out of place but I think a considerably big amount of people are forgetting the "Save Rating" button.

I just realized that my game has ‭367‬ plays and only 52 ratings. I think a lot of people forget pressing the "SAVE" button. I realized that I myself forgot twice too when I looked back on the comments I gave.

Then I saw Mark himself forgetting the save button during his stream. The audience had to warn him so he got back and re-rated the submission.

Can you make it so that on rating value change the rating is saved and remove the button.

leafoAdmin5 years ago (1 edit) (+1)

Good feedback. I don't know if I'll have time to fix this this time around, but I'll put it on my TODO list.

leafoAdmin5 years ago

I added a warning if you try to leave the page after selecting a rating without saving. Thanks again for reporting the issue.

borbwareSubmitted5 years ago(+1)

not really the jam page feature suggestion per se, but i'd like to be able to search jams by their name in the search box, not only games and creators! it's sometimes difficult to find an older jam from itch.io without resorting to google.

leafoAdmin5 years ago(+1)

Our search system is getting an overhaul and this will definitely be added!

JWoodrellSubmitted5 years ago (1 edit)

I think we kind of stress tested the itch system and broke it in a few places. for my two cents it should be sorted by fewest plays / ratings so that things will self level themselves

FumeshiSubmitted5 years ago (1 edit)

I submitted a game on Sunday and it 404s when anyone besides me tries to access it. Even people who have accounts tell me that my page is empty when they search my name. Was our submission deleted? I got it uploaded during the middle of the slowdown.

Edit: here is a link to the game's page. https://fumeshi.itch.io/robo-cross

leafoAdmin5 years ago

Your page is marked as "Draft" and is not published. It looks like you submitted it to the jam though. Maybe you accidentally unpublished it after you submitted?

FumeshiSubmitted5 years ago

Odd, it's saying it's released on my end. Where would I go to fix it? I'm sorry for my ineptitude, I just my account to sign up for this year's jam.

leafoAdmin5 years ago

"Release status" is a classification of how completed your project is. It doesn't have any effect on whether your page is visible or not. You can find the visibility settings on the bottom of your project's edit page. Hope that helps.

FumeshiSubmitted5 years ago

You saved my life. Thank you

VictorGladysSubmitted5 years ago (1 edit)

Games are often created as a team of developers. However, only the person that uploads the game and submits it to the jam gets a jazzy "Submitted"-button next to their name, even if the rest of the team is made contributor on the admin page. I do think that the button is essential when I post a comment on their game, for them to know that I too have participated. I circumvented the problem by adding the link to the game as my display name but well, it is not ideal ^^'

It seems like a small change that would make getting ratings easier, but of course I don't know the actual back-end implications of such a change.
With kind regards,
DaVici

leafoAdmin5 years ago (1 edit) (+1)

Nice catch. I'll try to fix this ASAP.

Also clever idea putting your URL in your display. I also want to make it more obvious when someone comments what their submission is to encourage cross ratings.

VictorGladysSubmitted5 years ago (1 edit)

That's lovely! And yeah, I was pretty happy that it fit just within the maximum of 40 characters :)

gardrekSubmitted5 years ago (1 edit)

I can't seem to filter the submissions by Linux support any more. I'm pretty sure you used to be able to. In the "Filter" box there's Windows, Mac, and even Android but not Linux. I assume this was introduced as part of this update, and is simply a bug or oversight.

Edit: For anyone wanting to rate Linux games having this problem, a work-around is to use the randomizer, which _does_ have a Linux filter, and as a bonus, can use multiple filters at once. Actually it'd be nice to have that functionality on the main list, too.

leafoAdmin5 years ago

I'm so sorry! This was a mistake from when I updated the code a few days ago. I'm ashamed because I'm a full time Linux user. The bug has been fixed and the filter has returned.

iwansquallSubmitted5 years ago

Several feature request:
1. Jump to specific number of pages.
At the moment itch io only have next or previous page, but sometimes ppl want to specific number.
Many other forum have this feature, but not itchio

2. Weird bug
I was using launcher earlier with randomizer, and I got some random Html5 game. There is no download button but if I go the developer link, and then the project, it shown at the bottom I can download the game. But that seem like window exe version. At least i expecting exe version to be shown on the randomize project page.

3. Rating game for jam in launcher.
Atm, when I finish 1 game and want to rate it for jam, it will open it in itchio launcher browser. I dont know if it possible to make so that you able to rate without needing to pressing back button to randomizer. Something like like pop up window. It look very differently from getting a random game and jam's submission pages (where you can rate on that page directly)

4. Different comment?
Apparently comment in randomize project page is not the same like in rating page. I thought I stumbled on to game that nobody have rated, then suddenly it have a lot of rating and comments. Ok, after testing the randomizer on browser instead of launcher. It seem like it bring us game's project page and not rating page. I will take note when im commenting next time so it will be on rating page rather than the projects itself.

Im done for now, should i write new post or edit one in case i have new suggestion?

kypelloSubmitted5 years ago

Hi, it looks like there is are two UIs at the top while using the randomizer

Sorry if you are already aware but I thought I'd mention it

JUSTCAMHSubmitted5 years ago (1 edit) (+1)

When you look at a jam submission page for the first time, the icon of the game a commenter submitted starts at the wrong size, then a frame later is correct. It just creates an annoying flicker as I'm scrolling through the comments, as each of the commenter's submitted games wriggle around whilst loading. Also, if there have been no changes made to your rating, can you grey out the save rating button? That way I wouldn't click it again out of paranoia.

gardrekSubmitted5 years ago

IDK if anyone has suggested this before, but I think that along with the other platform logos (the Windows logo, the Apple logo, the Tux logo, and the Android logo) there should be one representing web-playable. I suggest the HTML 5 logo. This doesn't just apply too this page but should really be reflected anywhere the platform logos are shown.