I'm sorry, while I fiddled my way to victory I missed several details. I was mostly looking at the man through most play-troughs.
Here's a picture to break this chain of misconceptions. We can ignore the top part as it's static.
First off, I was joking and referring to 2 and should have pointed to it from the side.
Secondly, I thought the purpose was to match the color in 1 with the value in 3. Of course, as you explained, 1 = 3. You should probably add some indicator of where 3 is and either leave it empty (so the player has to visualize 1 in 3) or maintain the same functionality. Either way is fine.
And third, I thought the down/up buttons had parenthesis and didn't notice they were arrows. Making those arrows wider would help.
So there it is, focusing on the bars instead of the man would make it clear. But I was watching to see how he'd react to me fumbling with the temperatures.