That' s mostly a normal thing in the game - it plays very much like solitaire, so when you lose, it's usually because there are cards left that you can't remove.
You don't have to remove all of the monsters (except on the last floor), so sometimes it's a matter of focusing on the important ones.
