Thanks for the feedback. The phantom does stay further away if you put down the glasses. (unless you specifically go towards him)
I suppose it could have been executed better, like automaticaly taking off the glasses on death or something.