Yes and no!
All the rating categories are used together to rank games in the overall rankings; so you have a list for each individual category (theme ranking, fun ranking, etc.) as well as an overall ranking. This means even if a game is amazing but had no audio it might lose out in that category and therefore have a smaller score in the overall ratings.
Games that get less than the median number of votes (which is 9 currently but might change) will suffer some score reduction, but otherwise a game that got rated 11 times and a game that got rated 111 times will both be calculated by the average of their votes from what I understand.