So essentially you need to do some matrix multiplication of the screen coordinates to convert them to world coordinates and visa versa. The camera in fauxton uses an actual 3d camera so you can follow along this tutorial to get the right coordinates:
I cannot recommend dragonitespams gamemaker 3d tutorials enough! If you are ever interested in learning full 3d in GM, he has a huge catalog of various tutorials for creating 3d environments and such in Game Maker!Viewing post in Getting the position of the mouse in a room
Great reference! I managed to get it working after a bit of tinkering. I ended up making my own simple raycasting funcion which assumes we are looking for an intersection on the z = 0 plane. Anyone reading this is welcome to steal it:
function simple_raycasting( dx, dy, dz, ox, oy, oz ) {
var _t = oz/-dz;
return [ox + dx*_t, oy + dy*_t];
}