It definitely does more, but as mentioned, without details on what files are in your portrait folder, and what all the attributes it's matching are, I can't really help you.
I did release 0.5, which has a debug setting. If you set that to true, it prints out a lot of details about how it's selecting portraits. That might help identify either what is wrong with my script, or with your portrait files. you can also go a step further and uncommon some files at the end of portrait.gd and it'll dump all attributes it's detecting for all portraits when it loads them. this can be a rather large dump. both of these slow the game down considerably when it's selecting portraits.