That's pretty much my only complaint about the game so far. 

I understand this not being a thing in multiplayer mode. But is there a technical reason why it isn't possible in single player?

this is because of the networking framework I use. In singleplayer, you are technically hosting a multiplayer game still, but it is not listening for any connections and is LAN only.

I see. Guess the only solution for now is rushing fights during dungeon runs or cast the recall spell asap.