I use custom made LoRAs for the characters along with the "reference" ControlNet to maintain their designs.
