You can say no, but ultimately, even if you get the other two on your side, which neither of them would want to say no to the first decision of a united army, you still somehow don't get the majority. This means you have to go with the duke's demands no matter what if you want the united army.
So you can't really refuse him while still doing the seemingly right options, meaning it's forced. I'm gonna assume it's a bug or something unintended.
