Thanks for the feedback.
Yeah the interactions with walls aren't ideal. I wanted to make them so that you have to avoid them to be faster. I dislike in some racing games that it is better for you to slide against a wall than take the corner properly. I wanted to avoid this just using stickier physics material - but that made it feel bad when the car jumped and landed on some part of the body. So the physics are actually quite weird and inconsistent.