The game doesn't tell you exactly what you should do, I was trying to press the buttons but only after a while I discovered you should hold them to progress. After a couple of failures, the fantasy screen overlaps the work screen so you can't see which button you need to hold, making it a bit annoying to try to find which button you're supposed to press while also avoiding monsters. And yeah, you need to avoid every monster and instead try to find the npcs to talk to.
I know I sound critical, but I really do think this game has some charm. The idea is great, it's the experience that's just buried under some UI problems and lack of tutorials. Overall, I personally did enjoy it. Well done!