Try after midnight. But you may need to rest at the fairy tree or use an energy potion to stay up late.
