There is two ways to capture an invader.
1. Fight them till they are weak of mentality or high on lust, then use a collar in the item menu.
( the chance depends on their mentality and lust)
2. Some has ability to capture in battle, so if you have a character you wanna use to capture an invader, then make sure it has the ability of capture.
Fun fact: You can still capture them when they on no health, as long you dont end your turn. Otherwise they just die at the end of your turn.