Yeah, i'm not exactly sure why its doing that. It could be something with the matrix functions GM has. I've never run into the issue. Does it rotate properly in the demo? Also are you scaling it by any chance?
Viewing post in Cube squishes during rotation
Interesting! I remember testing the x / y rotations and they worked initially, but that was v1, so maybe I've got some code weird in there. IDK why it would mess with it though since essentially all I am doing is setting a matrix for the rotations. Hmm... But yeah, sorry I can't help more! I will have to look into it when I have time. Fauxton is mostly un-supported for now (I've been very busy with other projects). It's best used as a starting block that you can then expand from. When I get some time I will try to look into it and see if I can pinpoint the issue. X/Y rotations with sprite stacking are less than optimal due to the fact you will see 'gaps' when rotated at certain angles. I highly recommend Stack3D as an alternative if you cannot get it to work, as I know 100% he has solved a lot of the issues I have not been able to. I also highly recommend this series on 3D in gamemaker if you want to learn a better understanding of how 3D in-game maker works. :)