Yeah that happens a lot! I think the game I shamelessly stole from (KanyeZone) handles it in some way to make that not happen.

My first thought of how to fix it is to make sure it's always X amount of degrees away from a 90° increment, so it's never straight