This is a really solid concept. I think it has quite a bit of room for growth; for example, why not have different elements of a room be impacted by different die (arrow traps, flowing water or wind speed). Each encounter could have unique things that are rolled for!