It works fine when I open the folder in Explorer and launch the executable there, but when I use the itch app, nothing happens. Itch still says it's running, and i can force close it.
Are you running on Windows or Linux? If it's Windows; could you run through .exe itself? If it's Linux; could you go to the folder and make sure it has write/read permissions, if so, can you try to run it via terminal by ./RetroArch-Linux-x86_64-Nightly.AppImage
I did not in the first place but it should've worked fine with both itch.io launcher and running directly. I've checked and I can run through Itch.io without issue. Could you ask Itch.io support if there is any issue with them?