I think the most important argument here is, that not voting is counted as a vote of some sorts. People do can see game descriptions and thumbnails without ever playing the game. So those voting players not trying out the game is also to be considered.
It can be argued how exactly it should be weighted, but I have no doubt that it should be done, if one wants to call it "fair". Even for small numbers.
Making people vote on a minimum number of games, is a good idea for various reasons, among it dampening the effect of having your mum vote on your game and your game alone.
And I agree with what leafo said about small jams and vote numbers. Whatever raw votings you would have, would be inaccurate to begin with. It is like measuring the thickness of a sheet of paper with a ruler and only having 5 sheets to measure. You need to do tricks like folding the sheets to have any resemblance of an accurate result.