arxiv:2610.40727

VoxLock: Target-Speaker Extraction for Voice Agents Without an Enrollment Step

Published on Oct 29

· Submitted by

Arbaz Siddiqui on Nov 2

Upvote

Authors:

Arbaz Siddiqui

Abstract

A fine-tuned causal target-speaker extraction model achieves near-commercial speech recognition accuracy on real recordings with low CPU usage and latency.

Generated by thinkingmachines/Inkling-Small

A voice agent's speech recognizer transcribes everyone near the microphone, including people other than the caller. I fine-tuned a public causal target-speaker extraction model (REAL-TSE, 27M parameters) on one GPU to work from a voice sample cut from the call itself, and released it as VoxLock. I score it by what a recognizer writes on real recordings. On Krisp's 265-recording benchmark it reaches 4.95% WER against 23.21% unprocessed and lets 9 background words into the transcript over 85 minutes of background-only speech. Its WER is not significantly different from Krisp's best filters and is significantly lower than the other commercial and open systems I tested. On 400 AMI meeting windows built and frozen before scoring, it cuts WER from 42.95% to 10.63%. The 8-bit model runs a live call on 77% of one CPU thread with 24 ms of algorithmic latency. Model, code and per-file scores are open.

View arXiv page View PDF Project page GitHub 88 Add to collection

Community

arbazsiddiqui

Paper author Paper submitter about 20 hours ago

Author here. VoxLock is a 27M-parameter streaming voice isolator for voice agents that needs no enrollment step: it takes the caller's voice from the first seconds of the call by itself. On the Krisp Voice Isolation Benchmark (265 real recordings) it scores 4.95% WER with 0.11 leaked background words per minute, the lowest of every model tested, paid or open (Krisp VI 2.5: 5.30%). It runs on one CPU core with 24 ms algorithmic latency and ships as a Pipecat filter, a LiveKit Agents plugin and in sherpa-onnx. Weights, training code and the benchmark harness are Apache-2.0.

Try it in your browser

O96a

about 17 hours ago

I want to be convinced by the privacy framing, but the eval doesn't get there — 4.95% WER against 23.21% unprocessed proves the extractor separates speakers, not that it works as a boundary. For that you need a false-accept rate: how often a bystander's words survive into the transcript, per hour of background-only audio. Nine words over 85 minutes is a starting point, except nine words isn't a unit — one leaked name and address outweighs forty "mm-hm"s, and a single averaged count buries exactly that.

The enrollment-free design is where I'd expect trouble in prod. Taking the profile from the call itself means the opening seconds are both profile and content, and openings are the worst audio of the session: "hello, can you hear me?" in wind, or the caller silent while someone else talks first. At 1.5s and half noise, does it pass everything through, or clamp down and drop the real caller? The 27M single-GPU footprint is what makes this shippable at all, so I'd take a documented failure mode over another filter comparison.