Bandit attitude does not affect the quest. I found what I was missing was after speaking to Jester I needed to just hang around at the track at night. It took me a couple tries because it didn't trigger on the first night. If this does work, you should be ready to infiltrate camp which I think happens in daylight. Don't even think about starting the fight without at least a couple dozen healing potions and maybe some restrain potions.