If you're using the starter sword, most enemies can be killed with one or two charge attacks. Perhaps this should've been more clear, but the staff is only supposed to be used in situations where it's very hard to use a sword, and thus staves have very low damage.
As for weapon switching, I'm not sure how to make it easier-- there's already scrollwheel and number selection support...?