The controller does work almost flawlessly with other games. There are a few hiccups here and there but I can usually fix it with some basic troubleshooting.
In regards to Celeste specifically, I saw that there was another version to install (FNA). I installed that and that somehow took care of it. I kind of just assumed that games would support some sort of input but I didn't consider it otherwise.
I'll check other games I'm having issues with and see if there's another version to install, but Celeste was the only one I've been able to get working so far.