Hey there, I noticed in your code that you send a lot of your packets (animations, messages, etc. ) through channel 2 which only gets read 12 times, this might be the reason why so many messages get lost or actors dont update.
A fix could be to keep 1 channel for messages and not limit this. I've tested it with a friend and it seems to help.
Also, with a lobby of 10 people, frame time is ~30ms. there's some functions you could simplify or even move to another thread. I'm not experienced with Godot, so I'm not sure what's causing the high frame times