R36 releasing on January (Public on February) will have rebalanced monsters. Hard mode is currently the most challenging, but not balanced.
Of course even in the rebalanced version, if you grind and train a lot, the following quests will be easy. That's by design, but the game has been rebalanced for a normal playthrough.
As for challenging monsters, normal mobs you encounter randomly are not challenging. Challenging monsters are only floor bosses. Floor 13 can be quite tough once you first reach it, but then it'll get easier. At that point, only quest monsters may be challenging.