The reason the nodes that you farm on your farm only give 1 exp is because a lot of people asked for the farm node exp to be nerfed early in the game development as it made the Agriculture Skill severely overpowered compared to all other skills as it both provided the player with exactly the materials they needed at the time and large amounts of exp without so much as leaving their house zone. This decision is not final though and may be rebalanced in the future.
Also, it is the level of the skill specialisation that needs to be leveled up to decrease the amount of VP that action takes (so Botany in your case), not the Skill itself, I made a mistake in the explanation sorry! pinocchio8400a's explanation was correct in that regard.