The term Adventure Game does not mean a game about an adventure. You can make an Adventure Game about building a ham sandwich in your kitchen. The genre is named after the game Colossal Cave Adventure. Games that are somewhat similar to that game are called Adventure Games.
It's much like the term Rogue-like. You would not call a game Rogue-like because it has a rogue in it. You call it Rogue-like because the game is somewhat similar to the game Rogue.
I hope this clears up the confusion.