Are you using the mono (.NET) version?
I had this same issue on the mono (.NET) version of Godot 4.0.2. I had all of my scripts in GDscript so I moved over to the standard edition and it has the web templates available.
Gosh yes, I was. Thank you. Now I have that weird Cross Origin Isolation & SharedArrayBuffer issue...
Just had to enable SharedArrayBuffer in the itch.io game page settings, now my game works. Thank you for saving me!
Happy to help.