What you described should work.
I currently see that your project is in “Draft” mode and not “Restricted”. Draft can only be viewed by editors (or by the special preview URL), and is not suitable to distribute your game.
I had to change it to "Draft" as that was the only easy way to share the link with a few people on Discord.
Before doing that I tested things more with a friend and I found out that it works as expected if you enable the option "download key can be claimed", but you get the 404 error page if you don't.