Getting rid of the slave mark and harness helped, but the guards in towns and banks still try to attack me.
After getting rid of the slave mark you need to bring back your morality to 50, otherwise you'll go to jail.