Itried uninstalling and uninstalling throught the itch app. Did NOT work, but adding it as a nonsteam game to steam, seems to have worked. If anyone else has this silly issue, try that.
When Steam is running, it overrides your system controller config by default, by launching the game through Steam it will use your in-game controller mapping for Steam Input rather than the desktop controller mapping.
Steam Input settings should be accessible via Steam settings.
I still can't make my controllers to work (Both Steam Deck physical controller or Xbox ONE controller). I added the game as non steam, launched it on steam as still the same problem.