This was disappointing to discover!
You can upload gifs, but these are significantly larger than the equivalent mp4. I was hoping I could embed a few short videos in my page that way.
I see something about a “video embed code” in the Video dialog, so it really sounds like you should be able to upload videos, too.