That's as feature, not a bug =)
I've got five keys, too many things, so many possibilities and lots of places to go. Which doesn't make it easy.
Yeah, I understand that the game is a little bit hardcore. But it's a lot of narrative objects, it can help. And don't forget to draw a map =)