I love the idea very much. However, as you said in the description, there is some balancing to be done. I got stuck on a slime with 45 HP, because both it and I had crazy high defence and not enough attack. So, I sat there for nearly 15 minutes and only once was damage dealt.
My bag composition was:
ATK: 3xd6, 2xd10, 4xd20
DEF: 5xd6, 4xd20
EDIT: This could be mitigated with critical miss/counter system where if you ATK roll is much lower than the enemy's DEF, you get damage instead.