I have met bread but I don't know if getting rid of the knight badge will lock him out even if you already met him, so will getting rid of the knight badge  prevent you from talking to bread even though you already met him?

Once you open the passage to the room he's locked in, it permanently stays open. Plus, any badge can open the passage, not just the knight badge.