TIL Box2D collision breaks, if you resize fixture shapes on the fly.
For me it looked something like this:
The bodies started to fall into the planet and occasionally teleport out of it again. I solved it by destroying and recreating the body every time it is resized.
If there is a better solution to this problem - I would love to hear it :)