I really like the idea of playing a meta-game to pick the next round of towers. The idea of that game being Yahtzee is interesting and it fits really well with the theme as a push-your-luck dice game. That said, it might help the gameplay to either tie the mechanics closer to Yahtzee, or to change the dice metagame to be more in line with the TD aspect.

• One interesting feature of Yahtzee is that you have to play each group once, so sometimes you want to roll a lot of 1's, because you can get an extra 3-4 points by the end of the game. This isn't the case in your TD, because even three 1's is useless when you can go for two fives or two sixes every round. However, I think a change in this direction would require a lot of changes to the tower defense aspect.
• You could keep the meta-game the same, but perhaps each die number corresponds to a type of tower. So maybe 1's are low-damage slowing towers, but four 1's slows a lot more than just one or two 1's. It seems like this would make each tower more useful, and I bet you could come up with a set of rules that are more interesting than Yahtzee as a result.