1. Is there a standard guide to what ratings should mean? (E.g. is 3 stars “meh, tolerable”, or “perfectly lovely but I’m not actively raving about it”?) Or, like in IFComp, an explicit absence of such a scale? Personally I tend to use a rubric similar to Sam Ashwell’s unless instructed otherwise, but I think that makes me a much harsher voter than many.
2. Is it better to play and rate as many games as possible (which punishes games I don’t like because of genre), or to focus on those I expect I’ll like (which skews ratings)?
3. Are there guidelines on how to interpret each criterion? I’m not sure “presentation” makes sense for, well, any text-based game. And how to rate “puzzles” in a puzzle-less game?

Yes, I know I’m overthinking this, but it’s important to be as fair to authors as possible — or at least warn them that judges may vote like capricious monkeys.

Related IntFiction forum thread

Hi Hawkbyte,

Thanks for your interest!  I have to be honest, we have no such guidelines i'm afraid!  It's a very good point though and i've added this to my list of "fine tuning" for next year.  For example, as you say, in my mind 3 stars is good, 2 would perhaps be "good in 1-2 areas but overall not for me" and 1 would be "unfortunately nothing here appealed or worked for me."

Thanks

Adam

[Disclosure: I'm an entrant in the comp]
Where I've come across something like being asked to rank puzzles in a puzzle-less game, I've been rating it 3 stars as "neutral": it neither has shiningly awesome puzzles, nor is it trying to do puzzles and failing. (I'd leave the score blank if I could, but I don't think itch allows that.) Others might not see the point of a game without puzzles and rate it 1, or be refreshed at not having to do puzzles at all and rate it 5. Their choice.

Note that (as I understand it) only the "overall score" ratings count toward the final results of the competition, so you needn't worry that you're unfairly lowering a game's score.