I feel the same way. Even when I was wandering I was having a good time. It always felt like a dream (which makes sense, it was in LSDJAM 2023). I personally am totally fine with the game being a visual novel and exploration sandbox, I just think some people didn't know that was the experience they were signing up for. I think that's mostly what it comes down to.
I also agree that personally a "skip" button would have made the experience worse for me.