In this experiment I used the Dreamshaper model that generates pretty nice illustrations. The ControlNet 'depth' alone, however, is not so great. As we can see it struggles with the details when the fingers are near the mouth and so the AI starts to create a horrible act of chewing finger nails.
Viewing post in (Beta) Multi-frame Video rendering for StableDiffusion comments
Looks great! I think the best method would be if you used something like blender and had it render perfect depthmaps instead of using the preprocessor, but I haven't actually tested that yet. I have heard a lot about https://toyxyz.gumroad.com/l/ciojz for that kind of thing but I have never tried it myself and I don't know how it would work for animations. I would like to implement the ability to add multiple controlnet inputs rather than just the guiding frames but the issue is that, as far as I'm aware, I can only change the "control_net_input_image" property to impact them all at once and I cannot set them individually with a script.
thank you, I will have a look at the Character bones thing.
and sd-webui-controlnet has an API that allows you to target individual ControlNet models:
https://github.com/Mikubill/sd-webui-controlnet/wiki/API
example is here: https://github.com/Mikubill/sd-webui-controlnet/blob/main/scripts/api.py
unfortunately that's for the API to standalone applications and not for scripts. There is a bit of a hacky way discussed on GitHub for allowing multiple controlnet layers to be controlled independently with a script, but I haven't tested it and I expect native support for it will be officially added soon enough anyway