There doesn't seem to be a lot of control over how self-voicing works, it's Renpy functionality that isn't really exposed to me (that I can see). I will look into it and see if I can come up with anything, but renpy was never really built to support a full on RPG and I've been facing that reality for a few years now haha.
So, I can't make any promises, but hopefully I'll be able to figure out... something. Sorry about the trouble so far, but glad to hear you've been enjoying the game otherwise!