I'll respond because i kept having a Missing Guys problem and figured it out: the sprites sometimes walk into your cursor as you're gathering resources, going about your business, which obviously means summoning a crowd to that exact spot. Very quickly that turns into trashing what the single sprite has. It takes only a couple seconds to trash an item, which often isn't long enough for me to notice AND save the item before it's gone.
It doesn't matter where you're clicking; as long as you're holding it, whatever is under your cursor is activating.