Yeah I should have made it clear that the projected face is the number on top if you move that direction, not the number on that side.
Oh damn. I just ended up memorizing the cube configuration (which is hard) and figuring out the numbers on the side myself. I though the projections were buggy. Good to know though :) Makes sense now!