I cleared it at level 48 on hard, so 80 is definitely not necessary. Tried restarting and building a decent character? Helps if you don't pick every fetish on creation or get any of the perks that boost fetish.
There are no difficulty levels. Also, what the bloody hell? The further you go, the stronger you need to be? What tragedy! Something completely unheard of in other RPG-like games [this is sarcasm, in case you don't get it]; things like "no level up/SL1" challenges could be possible on the likes of Final Fantasy entries and your average Soulsborne experience, not here. Flash news: your average RPG expects you to get stronger, or you get steamrolled even by mobs.
There are enemies that give you a decent amount of experience and that you can defeat in 3-4 hits, provided you take advantage of their weaknesses, this coming from someone who reached level 92 without farming stronger bosses, nor spending countless hours on this.
On character creation there's "difficulty" which changes how many points you get, how much spirit you recover on level up etc.
Hard makes all your stats start at 1 and gives you 20 to spend, with 1 sensitivity point, Easy gives you 10 points to spend, but all your stats start at 10, with 5 sensitivity points, making you much more powerful at level 1.