I LOVE the idea. Sort of a modern Harvest Moon. I think there's a huge amount of potential here and I really hope you go on to finish this game.
As mentioned, the plants wither way too quickly. I think the aesthetic is lovely and wouldn't suggest any changes. Holding Z to read text was a bit odd at first but I actually think it's a great idea - it allows people to choose their own reading speed in a simple and elegant way.