Really glad I went back and tried it again with that INT/DEX build! The first time I made it to floor 77 while avoiding as many enemies as I could, but eventually I was overwhelmed because I couldn't kill anything above the cultist/water monster level. Then I learned to equip the knife you start with. Also that the rest button exists. Then I defeated the Ancient One!
I think what really turned it around for me was the rest button, it completely changed how I played. Instead of avoiding danger as often as possible, it was actually the best idea to fight as many enemies as possible. You get more XP, sometimes items, and you can (to an extent) control how much sanity and HP you lose. It's also the only way to lower your curse level, so getting into fights is really the only way to win.