itch.io

How did you get the initial 6? They don't walk around in packs, and you have people on you from 3 different regions. I've done multiple runs, and most I've seen at once was two unless I run around and grab more. There are also red indicators when you hear people that show you someone is there so that you can avoid.
I understand that there may be an issue, but your screenshots do not convince me.
On your first one, you have three farmers, one fishman, one merchant, one traveler
On your second one you lost a farmer and found 2 fishmen. You didn't rest, but a fishman did. Even though you are 20% quicker than fishmen, they have no issues catching up to you.
On the third screenshot, you lost Peyton the merchant somewhere but found Stuart the merchant by going back. You also lost Tetrafin the fishman. You found Manley the farmer, a rabbit, and Weston, the traveler.  So from your second to third screenshot you shook off 2 and found 3 (I think we can safely skip the rabbit).

You do know that red indicators out of your view range mean someone is there by now, I hope, there should be no issue avoiding these people.

The last thing I want is for you to not have fun, so if you think the game is too hard you can change the difficulty in about 5 minutes.
In your game folder edit the "data/organization.json" file. Your starting location is 30,27,0 (x,y,z). In that file you can add or remove as many enemies as you wish. Here is an example:

{
        "id": 22,
        "level": 1,
        "name": "Fishmen",
        "map": "29,27,0",
        "workers": [313, 313, 313, 313, 313, 313, 313, 313],
        "recipes": [],
        "leader": -1
    }

Map is location, so 29,27,0 is directly to the left of your starting region. "workers" is a list of spawned entities during world creation. If you remove a couple of "313" from the list, you will have less enemies. You might want to edit these:

"name": "Travelers",
"map": "31,27,0",

"name": "Mansfield",
"map": "31,28,0",

After you make changes, make sure to backup the files, it may get overwritten by some update. Hope you'll have fun the way you want to. :)