You can't make file associations on windows without admin privileges. Imagine you "install" something hooking on .doc, that is recipe for desaster. Also, if you make installation, you need to make de-installation also.
But you might not need to. Make the user do it. First time they open your file named file.custom windows will ask how to open the .custom extension. Tell users to select your app.
I have two questions here.
Why use this method anyways. Having associated file extensions only have reason when you navigate to files with os browser. If you open files by opening app first, it does not matter. File extension is only shortcut for making os open the app to open the file.
And considering installation, itch app is supposed to install the games, but installation for most stuff is just unzipping it. Not even itch app itself requires admin privileges to install.
I for one would not want some recreational game or tool install system stuff on my system, like file extension. Not the least for the simple reason you stated. You are one-girl shop. You lack ressources to test your stuff.