How are you supposed to fish? Pressing E throws the line but does nothing else.
Edit: figured it out. The you need to press E again when the character is shaking. Not really intuitive, I thought their arms just got tired from not doing some fishing minigame.