it depends rly on where he is and if he is close enough to you that he can just get you.
one way to stop this is by warping him around a wall away from you then turn off your light.