First, the tech tree is the tentacle-looking icon in between the book and the note paper. It provides hints on how to advance your skills. Second, the combat system is a lane-based autobattler. Once you unlock some things and scout a territory, you can pick a "royal army" and invade other territories with it. Burn hearts to activate your monsters' special abilities. As for overpopulation, the only solution I've found thus far is to quite literally work my tentacles to death. Once they use up all their energy hunting, scouting etc, they become wounded and may continue to take damage until they disappear.
As for feeding, I don't think it's implemented yet, at least not in the public build. It looks like feeding is going to appear as an unlockable ability, like scouting.