I've tried to move around a whole lot thinking it'd be possible, but it never worked. Also, once the debuff stacks enough, you lose the ability to roll as well. I'll try testing to get them off again, but I didn't seen any results. I've even gone entire floors with them on me and never coming off no matter how much I move around. Debuff only going away once I've gone to the next floor.