Left click and hold with an axe to remove items.
I think an axe doesn't remove the water purifier and the cooking station for some reason...unless there was an update, idk.
i played the latest ?1.05 and its fixed :) 3/3/17