It's possible to take slaves from slavers or travelers without taking a reputation loss. Remember - word can only spread if there's someone left to spread it.
If you fight slavers, and Capture every slaver that's left, you won't lose reputation. Same for travelers - capture all of them, and no reputation hit. Then just quicksell the ones you don't want to keep.