I'm glad you're enjoying the game!
I feel like charisma is right where it is right now. Physical and magical builds are better for combat, while social builds are stronger at building relationships faster, which you can leverage to avoid undesirable confrontations. It also has a few situational combat actions that it's good for, and it helps to pass some checks during the story. Perhaps you should not focus only on charisma.