Some of the characters are more sensitive than others about Grimer's smell! Some if them will not play with you at all, or some of them will just need a short break.
There are certain gloves you can buy that prevent the smell, if you use them with Grimer! They are the gloves with the lightning bolt. (Or you can just not play with grimer, I GUESS)