Looks great! I think the best method would be if you used something like blender and had it render perfect depthmaps instead of using the preprocessor, but I haven't actually tested that yet. I have heard a lot about https://toyxyz.gumroad.com/l/ciojz for that kind of thing but I have never tried it myself and I don't know how it would work for animations. I would like to implement the ability to add multiple controlnet inputs rather than just the guiding frames but the issue is that, as far as I'm aware, I can only change the "control_net_input_image" property to impact them all at once and I cannot set them individually with a script.
Viewing post in (Beta) Multi-frame Video rendering for StableDiffusion comments
thank you, I will have a look at the Character bones thing.
and sd-webui-controlnet has an API that allows you to target individual ControlNet models:
https://github.com/Mikubill/sd-webui-controlnet/wiki/API
example is here: https://github.com/Mikubill/sd-webui-controlnet/blob/main/scripts/api.py
unfortunately that's for the API to standalone applications and not for scripts. There is a bit of a hacky way discussed on GitHub for allowing multiple controlnet layers to be controlled independently with a script, but I haven't tested it and I expect native support for it will be officially added soon enough anyway