Certain effects like shrinking the chest to lvl 0 will be needed to get to certain areas (i.e. fitting through doors), but most content will be avoidable.
You'll usually be given context about what is going to happen before it happens so you can choose to continue with it or avoid it.