"The part that is not under my control is Unity’s physics system, which could have something to do with the issue."
Yes, Unity's physics engine is not deterministic. That would be the source of the behavioral not being deterministic since the behavior is dependent not only on the neural network but on the inputs to the neural network which in this case may vary from one run to another due to the physic engine not being deterministic. Personally, I like it that way, because it gives us a look at how the neural network reacts, rather than simply how it once reacted.