You are supposed to lose the first time, and if you really want to win, ask Nighveil.
NO, he said:
"Those guys at the big table over there. They are the guys you are looking for. their boss is the one you'd want to squeeze information out from, the bald guy."
Nothing about winning, and if you lot to them you should be getting an item called Bandits' Hideout Location already, which will help you progress the game.