We run one for every game we make - they're very worthwhile! Once you've got the core loop or mechanic of your game established, or a fairly representative version of the UI, ask five to ten people who've never seen your game before to play it with you watching them. Don't try and guide them through anything - just watch and see how they get on. You'll find that patterns in their behaviours start to emerge, and that's when you can start to figure out what is working and what needs changing.
As an example; when we made Village Cricket we have a state where a player can lose a match and is then presented with 'Retry' or 'Return to Menu' buttons. Due to the way the buttons were positioned we noticed during the beta test that a lot of players were accidentally hitting 'Return to Menu' straight away, and then cursing themselves because they'd meant to have another attempt instead. We swapped the two button positions over, ran the test again with a couple of other people, and nobody had any problems anymore.