Yeah I do agree, I think that would make the game overall more fun. After the jam I thought about the idea a little more and I think making the objects behave more like something you can push and then pull would probably have worked better (kind of like the game Control in a sense), adding that on top of the sword working as a magnet would definitely make the game better.
Thank you for the feedback!
-Thomas