This forum thread might be helpful:
https://forum.wordreference.com/threads/matter-of-degree.3240353/
This post in particular:
"It's a matter of degree" means "It depends on how much..."
It's raining? That's really bad.
Well, not necessarily, because we needed some rain for the crops. It's a matter of degree: too much rain at this time of year would spoil the grape harvest.