itch.io

One way to fix this would be to change the type of the colliders. Make them Trigger colliders. If you're using raycasts to detect the clicks, raycasts can be told to detect trigger colliders. If you're using the OnMouseDown method, then you may not have to change anything else at all - though I haven't tested this myself, so you'll have to play around with it.

I use tons of trigger colliders in my games. Once you get comfortable with the difference between regular colliders and trigger colliders, you can mix and match them to do some extremely cool things.