That's what I deserve for trying to fix a potnetial desync issue in the last 10 min...
I removed the fix code and now should work fine (although this might mean rubber banding for the local player after a while, still better than freeze, I need to work on that more and make a tutorial on all the issues and stuff I had)