I am absolutely the same way. I think it is really tricky to judge the difficulty of the game you are making, because it is always going to be easier for you to play then others. That's in theory why lots of play testing is critical, but I usually don't have the time (or the people) available to properly play test. If this helps at all, I ended having it so that the player can continue in the game after playing each level, regardless of how well they did or if they "died". That way everyone still can play through the game even if it is too hard for them. Anyway these are all things you can keep in mind for next time.
(My game if your interested: https://itch.io/jam/brackeys-4/rate/722009)