My personal observations tell me, you usually know exactly why a comment was downvoted.
The problem if we wanna call it one, is the interaction willingness of people. Not everyone bothers to write replies. Actually, most people do not. And if you look at some of the discussions that spawn if two people meet that do bother, it often turns out awful. The comment section should be about the game, not about having an escalating discussion. And it would explode if all those 20 people up and downvoting would take part and make replies and comments themselves. It is the comment section, not the discussion section. And I would say, up and downvoting is usually appropriate to comment a comment.