I think you're confusing Adventure Game with Action-Adventure game, a completely different genre of game despite sharing a word in common. For instance, your example, the Legend of Zelda, is not an adventure game.
Adventure games are story focused by definition.