Your idea is great and I think you could make it more confusing while keeping it's playability.
But in this jam I found out that it's really hard to make a game confusing yet playable. You did a nice job and made a really interesting game, but I think it could have been made more challenging for this specific jam.
BTW, well done :)