Yeah, the latency is a bit of an issue. It's interesting how latency creeps in and gradually gets worse as the game goes on. It's perhaps why we haven't seen mic input used widely in other games.
I made my game in Godot and I got the feeling that the live audio input was coming in separate to the game's tick function, as when I paused the main tick process to restart the game, I noticed that the audio input got really out of sync.