It appears that your racial bonuses to mental stats are applied every time you "increase" them. I put quotes on increase, as I noticed it when trying to increase a 100% Nereid's courage and her courage was dropping instead, -9 points when using +1, -5 when using +5. 100% Nereids receive a -10 penalty to courage with the mod, so it would appear it is being applied to their stat changes and not just the total. After noticing it there, I also noticed that it applies to positive stat bonuses (I had a 96% Demon/4% Orc that went from 48/65 to 73/65 Charm off one +1, and then to 89/65 (her cap) on a second).
It's not elegant, but I've at least found a hack to fix it.
In globals.gd I added:
var modstatdict = {sstr = 'str_mod', sagi = 'agi_mod', smaf = 'maf_mod', send = 'end_mod', cour = 'cour_racial', conf = 'conf_racial', wit = 'wit_racial', charm = 'charm_racial'}
Then, in slave_tab.gd, I modified the first line of both func mentalup(mental) and func mentalup5(mental), to read:
person[mental] += 1 - person.stats[globals.modstatdict[mental]]
and
person[mental] += 5 - person.stats[globals.modstatdict[mental]]
This is basically subtracting the racial bonus when improving the mental stats, so not exactly ideal, but I couldn't find where it was being added so this is the best solution I could come up with. It shouldn't effect anything elsewhere either, since it's contained within the function.
I don't think you need that workaround (honestly not sure what other effects it might have)
the only place it gets added is in expansionsetup.gd (last function):
func setRaceBonus(person, increasestats) (don't look unless you want spoilers on the hybrid recipes)
the function is only called in a few very specific places though. Should just be when an npc is conceived, generated (plus special cases for player and starting slave), or created through Slime breeding. The code for the races is in there. Pure Nereid gets: bonus_courage -= 10 so negative courage starting out is possible. If she had -10 and you added 1 point that would give you -9 courage. Also, there is a known bug in Aric's that causes jumps in the displayed vs actual mental stats that are fixed once you start adding to my understanding (and I've seen it lots of times in gameplay). I think that's what you're seeing in that Demon/Orc hybrid.
The Nereid actually went from 48 to 39 to 30 to 21, etc. It was dropping by 9 every time I pressed the button. I tested over several characters (on multiple saves/games), and all of them increased or decreased in the stat by the racial modifier +1. It may just be an issue with my install or something, but it's working fine now.
The cause of the undue increases in mental stats that he is seeing is the result of changes in person.gd. cour_get, conf_get, wit_get, and charm_get were all altered by Aric's mod to return the base amount plus the racial amount. mentalup and mentalup5 both use += on the getset stats rather than the raw stats, which results in the hidden addition of the racial bonuses. This problem should not be exclusive to the manual increases in mental stats and will likely require a fair amount of work to fix.