Did you remember to equip the net before starting a battle? You need to open up the menu, select equip, and add the capture net as your tool. You also do that with your fishing rod.
