Yep, there was a step I forgot. I forgot to download the Microsoft XNA framework that was also on the download page. Hope this helps anyone else who forgets!
Simple as that. I downloaded the windows zip, extracted it into its own folder, and clicked celeste.exe. Nothing happens, even running on administrator mode. Is there a step I'm missing? I'm running Windows 10 32 bit on my older laptop. Anyone else having this problem?