When trying the fishing mini-game if you hit Z or Space to move to the minigame you auto-fail it straight away. It is worth noting that the follow-up dialogue that says you failed can be advanced without issue even if you do press Z or Space so it doesn't have a huge effect. (All 3 "starts" for the key fishing mini-game have this issue the picture is just of the first one)