I'm noticing that if I download and install larger packs of clips, the audio clip selector prioritizes content from those larger packs.
Perhaps the algorithm could select a pack first, then choose a clip from that pack or one of the sub-packs inside so you're still likely to see smaller voice packs without disabling the bigger ones.