Oh, I think it's deselecting them automatically if the tokens don't correspond to the card. Looking back at it now the UI is really unclear.
Thanks for checking the game out!
