You can use salt on the ghost while wearing the purple necklace to take stats back from him.
You can hold off during the night, but it might be unlikely when you are really big. Meditating multiple times helps. Also, 2 updates from now there will be some more things to help with holding off during a pact