Really cool concept - I love the idea of the stories that are brought back affecting the Library in unpredictable, organic ways, like a real plant. How you treat the world while on adventures, and what you say about the world (and what people say about you) all affects this literal physical place you're growing together as a community. If your plant is becoming spiky and carnivorous, maybe look in the mirror and consider what you've been feeding it.

Strong Borges vibes!
Also a bit of Stygian Library/Gardens of Ynn?