I may have found a solution.
Based on this post: https://forums.rpgmakerweb.com/index.php?threads/movie-not-playing.50798/
You need both MP4 and WebM files of the movies to get movies to work on web version of RPG Maker MV. The MP4 also need the correct codec: h.264. This can be done with a video converter.
(It seems like MP4 does not work in the editor though.)