Congrats on your first game jam! This is the beginnings of a very interesting idea, and I had an 'aha' moment when I noticed the environment change.
I think the mouse buttons as a story telling mechanic is a fantastic idea, I just wished the story was a bit more fleshed out / there was more to discover.
Great work!