Second this idea. You have a little plus icon next to the tag to add it to my existing tag, all you need is a minus as well. Can't imagine this being terribly difficult to implement. More complex operators like and/or could also be helpful.
