Can I use RightVoice commercially?

Yes. The data is CC BY 4.0 and the code Apache-2.0. Cite it as RightVoice (Crele) with a link to this page.

Crele made RightVoice and is in it. Is it fair?

The data, the build code and the scorer are all public, so anyone can check the numbers. Scores use only the output audio, and the open systems run at their published settings.

Is there a RightVoice leaderboard?

No, you are welcome to use the benchmark for your own evaluations.

Why is SI-SDR not in the RightVoice table?

It measures waveform match, not what a listener or a transcript gets. It is in each item's scores.

Does Crele train on RightVoice?

No. Its readers, rooms and noise clips stay out of every Crele training set.

Benchmark · v1.0

RightVoice A benchmark for keeping one voice from a conversation

RightVoice is Crele's open benchmark for keeping one voice: given a conversation and a few seconds of one person's voice, a system should return only that person.

Code on GitHub Data on Hugging Face

Which systems keep only the right voice?

System	Target words lost	Wrong-voice words per 100 target words	Wrong-voice words per minute	Words per minute, target silent	Words per minute, target absent	Activity F1	Voice match	Quality
Crele Focus 1.1Speaker isolation in the Crele app	33.9%*	3.7	2.98	11.2	2.0	0.834*	0.699	3.48
USEF-TFGridNetResearch target-speaker model	43.3%	24.5	22.76	48.1	65.9	0.799	0.570	2.90
DeepFilterNet3General speech enhancer	79.7%	56.2	47.35	85.8	69.5	0.738	0.766	3.39

RightVoice v1, seed 20261022, 1,530 items. Lower is better for the first five columns, higher for the last three.

* From a test version of Crele Focus 1.1 with a different check for whether the sample's speaker is present, set before RightVoice was run. Every other number is Crele Focus 1.1 as released, which loses 39.4% of target words with an activity F1 of 0.782.

More information: README.

What does RightVoice measure?

RightVoice scores three failures apart: lost target words, words from other voices, and speech while the target is silent.

Target words lost: Word error rate against what the target said.
Wrong-voice words: Words from other voices in the output, per 100 target words and per minute.
Talk over silence: Words per minute while the target is silent, or absent from the recording.
Activity F1: When the output speaks, against when the target spoke.
Voice match: Speaker similarity between the output and the clean target.
Quality: Predicted listener score, DNSMOS P.808.

What is in the RightVoice set?

Sources and licences

RightVoice v1 holds 1,530 two-person conversations, 16.45 hours of 16 kHz mono.

Voices: LibriTTS-R test and dev readers.
Turn-taking: AMI meeting timings: overlaps, back-channels, long pauses.
Capture: Rooms, FSD50K noise and music, a TV or phone voice, gain drift, clipping, Opus, AMR-NB, G.711, packet loss.
Each item: Mixture, voice sample, clean target, other voices, text with word timings.

How do I score my own system on RightVoice?

Download RightVoice, run your system on every item, check it on the quick subset, then score the full set.

Step 1

Get the code and the data

Python 3.11 and uv. The download includes the reference systems' scores.

Terminal

git clone https://github.com/Crele-AI/rightvoice && cd rightvoiceuv sync --extra scoreuv run hf download crele/rightvoice --repo-type dataset --local-dir data/build/v1

Step 2

Run your system

One WAV per item, named by its id, at any sample rate. An empty WAV returns nothing.

Files

data/build/v1/items/<id>/mixture.wav   input: the conversationdata/build/v1/items/<id>/sample.wav    input: the voice sampleoutputs/<id>.wav                       your output

Step 3

Check it on the quick subset

192 clips, about a tenth of the set. A check, never a substitute for the full run.

Terminal

uv run rightvoice items --build data/build/v1 --quick > quick.txtuv run rightvoice score --build data/build/v1 --system my_system --outputs outputs --quickuv run rightvoice report --build data/build/v1 --quick

Step 4

Score the full set

95% intervals against every reference system. About 2.5 to 5.5 hours on one 24 GB GPU.

Terminal

uv run rightvoice score --build data/build/v1 --system my_system --outputs outputsuv run rightvoice report --build data/build/v1

Runners for the reference systems: README.

Questions

Can I use RightVoice commercially?: Yes. The data is CC BY 4.0 and the code Apache-2.0. Cite it as RightVoice (Crele) with a link to this page.
Crele made RightVoice and is in it. Is it fair?: The data, the build code and the scorer are all public, so anyone can check the numbers. Scores use only the output audio, and the open systems run at their published settings.
Is there a RightVoice leaderboard?: No, you are welcome to use the benchmark for your own evaluations.
Why is SI-SDR not in the RightVoice table?: It measures waveform match, not what a listener or a transcript gets. It is in each item's scores.
Does Crele train on RightVoice?: No. Its readers, rooms and noise clips stay out of every Crele training set.

Why we built it: Introducing RightVoice

Try Crele Focus 1.1 free