i believe that was the original plan but after some... issues involving people nearly finding him he decided to leave them in someone else's care so no one would find him again and now his army is only of those he trusts/tolerates. so they kinda just exist now and your a babysitter or something.