My character's charm factor's already four even without upgrading it. Would have thought that'd be enough, but still not working.
May try getting factor to 5 just to see what happens, though the mats for that are a pain in the neck :) Things we do for a princess' love... :D
Updated : Nope, even with factor 5 it's failing. Gotta be something wrong somewhere if you managed it at 4.