I discovered that removing the background helps with maintaining consistency. I'm using this extension:
https://github.com/AUTOMATIC1111/stable-diffusion-webui-rembg
A short video about it:
