Sorry for the second reply, but I can't seem to reproduce this on either floor that has the bandits. Did you find both switches on the floor already? That's the only thing I can think of at this point.
Asking cause the bandit's switch on the poison floor opens the second door.