Are you using rigidbody physics for the character movement? I recommend either a faster acceleration with high drag, or swapping to a character controller for snappier movement. Basically, the movement should not be something that draws the player's attention away from the rest of the gameplay, because it seems like you want to experiment with exploration and risk/reward. If the player becomes frustrated with the movement, they are unlikely to enthusiastically engage with the rest of the game. That's just tuning though, so it shouldn't be too hard to play around with and come up with an improvement.