It seems like it just didn't work with the model I was testing - the Nova Anime XL.
I will investigate, but it doesn't look like it was a matter of the settings I had on that model.
ah yea I don’t think controlnet works for SDXL models(on ai horde)