I don't think either of those are the issue as much as simply the controls needing to allow for more player expression. At the moment it feels like the only real way to express skill is through the timing of the double jump, which can work, but due to the fixed jump height and jump arcs often just feels like the game isn't as responsive as it should be. I feel like it's probably just my personal preference and it's likely not the right call for the type of game you're making, but going back to the example of things like JK and GOI they both allow for an incredible amount of player expressiveness in setting up the jumps even if you're the mercy of physics once you are in the air.
There is also the other factor of the game properly recognizing collision, often if I hit the corner of a ledge the game would get stuck in a falling animation where I couldn't move, grab the ledge, or jump off the wall; but if I let go of the movement command I would then get locked into a freefall and just have to eat the lost progress rather than simply grabbing the ledge below like normal.