I think change the friction of the object a bit, i had the same problem and I fixed it by changing the friction of the object and moving the position around the colliders, I dont remember the exact thing which I did but it was probably some collider or game object issue for me atleast.
Or try rigid body collision ?
Maybe put a function or something that as soon as it detects a collision the collided object should go front or something or maybe a force
Idk I am just giving ideas it may or may not work out, probably try to ask on unity's discord if this doesnt work out