Is there a way how to remove texture from a custom material?

Yep, click the same texture again to remove it. If you're unsure which texture you've used, click a random one twice.

Thank you.