The problem with the difficulty scale is that in order to overcome it you have to grind - and it's horrible. The most efficient way is to spend, literally, days in a mine going back and forth until you can get your stats up. If not, it's somewhere else that's equally back and forth, just not as efficient (fishing, fletching, for example). There's really no fun gameplay way that helps you increase your STR/INT/DEX/CON and gives you a sense of game/story/character-development progression as you do it.
(Boss rare-drop grinding helps a little but lots of people don't care to do it and would rather just buy the item and so they do (using money gained from their mining), whereas those that do try are not adequately rewarded because the EXP increase for minion and boss kills is minimal compared to being in the mine.)
The average casual player has thousands of other games to choose from so why would they want to grind in this way when it's not fun and not fulfilling? Hence the dropoff.