Yeah that sounds like a good design choice, I guess it seems better if it was stereo all the time but actually in gameplay might be too easy.
We also thought about reflecting audio like in real life but that would be to much coding haha. Overall thanks for the feedback!