If I had a guess most popular looks at the amount of times people played it, where most rated simply looks at how many times it has been rated.
So, a "crappy" game, or a game that has no value in being played again may end up with a lot of ratings. However, as no one or few few will play it again it won't rank high in the popular category. A game that people enjoy and can play again will probably have people play it multiple times. When people do that the game gets seen as more popular and thus will be sorted higher, however as those people have already rated the game those don't change.
So, in short
Sort in rating-> How many people rated it
Sort in popularity -> How often it gets played
Disclaimer: This how I think it works, purely based on the names. I did not look into it at all.