The raw scores = sum of all votes (divided) by the number of votes.
If the votes are below the median, then the raw score gets decreased depending on how far it is away form the median votes. Resulting in your socre.
This is to prevent a game that gets only 1 voter with full 5-stars to win the competition.
To make sure that low vote count is less of a problem I would suggest that GBJAM also allows voters from outside of the competition to vote, just make it so that those outside-votes only count as 10% of a vote and that the total of outside-votes can only count as 25% of your final voter count (this is to prevent score boosting using outsiders).
This would allow for more voters to make the voting more equal (as in, less decrease due to low vote count) AND it would fetch more publicity for GBJAM.
Everybody votes in a different way, so overall it will work out, but still, the correlation between Gameboy Feel and graphics is still very much there. Perhaps the category can be explained better to help voters.