So it's now working on desktop but not on phones :/
Viewing post in Percussionanza jam comments
After some digging I found this https://caniuse.com/sharedarraybuffer it seems quite recent, except for chrome which got it in 2018. Not using it in my case throws everything out of sync :(