Because I don't trust Unity's physics (by which I mean they always find a way to act differently than I expect) I'd suggest resetting the ball holders first and possibly instantiating a new ball so that it's not secretly carrying over any momentum. I might be overly cautious though!