Wow, thank you so much!
To answer your question: Basically, every affected Rigidbody gets their own gravity direction based on their position with the center of the circle world being the source of it. I just calculate the direction from the center towards the position of the object. When they move I create a move Vector that has a positive or negative value on the x-axis (think of it as local move direction), then transform that into a world direction (so that their movement is based on their rotation) and set it as velocity. Then I add, when the object is not grounded, gravity direction + a force (speed) to their velocity.
It's probably not the most elegant or performant way but it works I guess. I can send you some sample code if you're interested, just message me.