Fun concept! I've seen it executed in 2D-tiled games though not 3D, but it did end up translating quite well once I got the hang of it.
Some quality-of-life improvements like animating the camera rotation, rotating the controls with the camera, and a less jarring "can't move" sound effect, would have definitely improved the experience.
Interesting game overall, and now I'm left pondering the mind-breaking levels this setup could open up to...