Honestly with Unity I've never bothered with their navmesh stuff - it was 5 years too late, and way too weak for what it does. Much better to roll your own for simple cases (like this) or get one of the assets that does it far better I think (e.g. the A* free assets that have been around for 15 ish years now)