You have more patience than me. I tried doing something similar to that like year ago and getting frustrated with that was what prompted me to start getting more familiar with vertex buffers and 3d space stuff.
Since you're using draw_sprite_pos() functions, does this only look right with flat shaded sprites? Like would a brick texture have shearing issues?