Hm? I'm pretty sure as long as you route around it there shouldn't be a quest that's directly tied to the White Oak. If you are talking about the Darkness In The Woods quest, that's for a Low Demon that's further back in the forest. White Oak's pretty dang strong (and not necessary for progression), I wouldn't really recommend fighting him unless you can 1 or 2 shot the saplings with a critical Slash.
I haven't gotten that quest yet, I was just trying to explore side areas before progressing with the main quest. Good instinct as it turns out, I found another star metal. I don't think I can take on the demon or the tree yet, though... is there anything else I need to do before crossing the big scary point that says "This will irreversibly progress the story"?