Agreed!
I'll let you in on a secret: What you suggest is exactly what I was trying to do, but 3D math under time pressure didn't really work out for me, so I made an executive decision and went top down instead, not because I thought it was a good idea, but because that was doable. I just wish I made that decision much earlier, then maybe there would have been actual gameplay.
Lesson learned: Don't do 3D in a game jam if you haven't ever done it before. It should have been a 2D game from the beginning.