Gotcha! You can right-click to clear the item you're holding, but it's odd the broom isn't clearable in any other way.
I'll get on that.