I have a 1600x1660 monitor and an ultrawide monitor and there is no option for anything that fits and borderless doesn't really help. Is there a way to put in a custom resolution?

You can force any resolution through the GameData.dat file.
But there might be issues, since the game is oriented for 16:9 resolutions mostly