No, you shouldn't have.
There are three tree monsters which appear on the world map.
They all appear during the game's event triggers, especially if Innis told you to go after them.
Did you go into the cave and defeat the boss within the fairy swamp
Yes I defeated the boss then immediately returned and purified the area after, I returned to town and slept. Then got attacked by the assassin and then slept til dusk. Talked to Innis got told about needing another oak. And now have scoured the map looking for one. I already got the ones located just outside of the village and the one by the druid's forest area. It seems the Oak that is east of the swamp is missing?