There's a fine line between genres and subgenres. Visual novels do count, but only barely. Same with some hidden object games.
It's hard to explain Adventure games to people who haven't played them. Those who have played enough of them tend to get a general feel of what is an adventure game and what isn't.
An example of a game that is mostly a visual novel, but also has elements of adventure games and rpgs is Leviathan: The Last Day of the Decade.
As for why not everyone would be doing visual novels is because there are some of us who joined this jam specifically because it's for making adventure games.