You can't take it off (it doesn't take an equipment slot, but it stays in your inventory permanently). It unlocks a series of mind control scenes around the game. You can't avoid the scenes by any means other than avoiding the NPCs themselves who have them (you can't really know beforehand who can trigger these scenes).
If you don't like being mind controlled, definitely avoid the charm.