Thanks for the feedback, I noticed that issue as well, I think its because the raycasting requires you to be looking directly at the object, when I play tested I could blow everything p fine but some items definitly took a couple tries.
If your interested, I'd suggest creating/activating a collider in front of the player when they press the Destroy button. Could destroy/deactivate it after 0.1s say. Might be more reliable than using a raycast.