It is a lot of work, this is like the 7th "engine" I made over the years, it's probably the best one yet, but there are still a lot of things I'd love to do different, but then I would never manage to actually make a game with it.
For the VR support I mainly just implemented a "VRWindow" from my "window" class (which represents a single application window on the screen). This VRWindow has a "swap chain" (which has a couple of buffers) just like every other window which can then be rendered into. The only difference is that instead of this being backed by a window on the screen, it passes this on to the different VR SDKs which all have an implementation of this window. The VRWindow also has some additional methods to query input, head position and set haptics.
I then have a VRCamera which does some of the more general camera setup, creating two cameras attached to a head which render into a vrwindow.
Actually this now looks more complicated than it really is :D The result is that I can use the same code for the different VR headsets / SDKs. The VR part was pretty easy to do, but I don't have support for some of the optimization stuff yet like drawing both eyes at once with instancing or foveated rendering. Both will need some adjustments to the rendering.