Looking into this. I've occasionally experienced this, but not consistently. Usually a restart resolved it. If it consistently occurs for you lmk because I may want to get some debug info from your build.
I will also say that based on recordings I've done, any time I've had it running for more than 3 minutes it starts to desync rather suddenly. When this desync is occurring I've noticed a sizeable delay between when I speak and when the mic gain bar registers audio input.