There was a phrase with a 2 for sweatdrop face, and I had 0 sweat drop faces on the right side bar, but choosing that option made it go down to -4. Also every icon has it's own bar of faces.
On paper this game sounds fun, it's a pretty great premise, and I'm sure there's some actual math going on, but I just can't figure out what the UI is trying to tell me.