No, you cannot rename Maelstrom.exe, it won't run because it checks the app name and it won't be possible to run with Steam overlay because my understanding is that you can only do that with games available on Steam. So your strategy to purchase COAS with START.exe and copying Maelstrom into the Steam directories and renaming won't work.
Viewing post in Running COAS with MOD through Steam.
Hi, kind of on this topic. I have managed to get ERAS to work but just downloaded GOF 2.5, I've completely forgotten how to install it all etc and my copy of COAS is on Steam. Can somebody give me a simplified walk through please. I have copied the ERAS files into a separate folder and re installed COAS fresh. ERAS still works with this setup in play just trying to get the same setup with GOF 2.5.
Do not start with a copy of COAS from anywhere else like Steam, or GoG, or from cd, some of the scripts/resource files are incompatible; everything for GOF is already in the Itch link for that version.
Download GOF from the Itch link, then point a shortcut to Maelstrom.exe as Target and the GOF 2.5 folder for Start In, or copy the following files directly into the GOF folder and run it direct from the GOF 2.5 folder:
Buccaneer's Reef
Config.exe
fmod.dll
Maelstrom.exe
Maelstrom.exe._lc