This happened to me every time I left the game idle for a few minutes, once to try to fix the audio in my stream, once to get a snack, and once to pet my cat. So frustrating.
