Introducing Crele Nimbus: an open model that hears who is speaking and what is happening, live

What is Crele Nimbus?

Crele Nimbus is a real-time audio decision model from Crele. It listens to a live stream and answers your yes/no questions about it every 40 ms, with an event each time an answer starts or stops. Two answers are built in: whether the user is speaking, and whether someone else is.

The code is on GitHub and the weights are on Hugging Face, both under Apache-2.0. The Crele Nimbus page has the reference.

How do I make a voice agent ignore the TV?

Stop the agent on the user’s voice, not on any voice. Voice activity detection hears that someone is speaking, never who, so an agent stops for the television, answers a bystander, or waits out someone else’s sentence. Crele Nimbus answers two questions instead of one:

• Is the user speaking? The voice you enrolled, from a few seconds of it.
• Is someone else speaking? Any other voice: another person in the room, or a voice from a TV or a loudspeaker.

Both are answered every 40 ms beside your own questions, and both can be yes at once, so you can see when someone talks over the user.

How do I detect barge-in from the user only?

Enroll a few seconds of the user’s voice, then stop the agent on the user signal’s started event. That is personal voice activity detection, live: the TV or a colleague starts the other signal instead, and the agent keeps talking.

session.enroll(user_voice_16k)       # a few seconds of the user for update in session.push(pcm_16k):  for event in update.events:      if event.question == "user" and event.kind == "started":          agent.stop()             # the user interrupted

The enrolled voice is a voiceprint, which is personal data in many places: get the user’s consent, and store it as you would any personal data.

How do I know the user stopped talking while others keep going?

Act on the user signal’s stopped event. Another voice does not hold it open, so the user’s pause is seen even while someone else is still talking. stop_secs sets how long the voice has to stay quiet, 0.3 seconds by default; raise it so a breath mid-sentence does not end the turn. It hears when the voice stops, not whether the thought is finished.

How do I detect music, laughter or a siren in real time?

Ask in your own words. Each question gets its own answer every 40 ms and its own started and stopped events, and you can add, change or remove questions while the audio streams. Questions worded in ways the model never trained on scored 0.892 against 0.911 for its own wording.

session.ask("music", "Is music playing?")session.ask("laugh", "Is someone laughing?")session.ask("noise", "Is it noisy?",          yes_means="loud traffic or machines", no_means="silence")

Is there an open-source Jev for audio?

Crele Nimbus answers Jev-style yes/no questions about live audio, under Apache-2.0. Jev, TypeSafe’s decision model, reads text and does not take audio, so with Jev the sound has to become a transcript first, and a transcript drops the music, the siren and whose voice it was. Ask Crele Nimbus about the sound and Jev about the words; the side-by-side comparison is on the Crele Nimbus page.

How fast is Crele Nimbus?

Crele Nimbus says yes about 0.2 seconds after a sound starts: a median of 0.24 seconds over the sounds it caught, on 500 real AudioSet clips with the default settings. For speech it is 0.21 seconds, with 97% of 181 speech onsets caught.

It runs in real time on a laptop CPU: it keeps up with live audio on an Apple M4 Pro with 8 and with 32 questions, because the audio is heard once and every question reads from it.

What does Crele Nimbus not do yet?

• Its questions are in English.
• It follows one user voice per session.
• The first second of every stream answers “can’t tell” on every question, while the model gathers context.
• Crele Nimbus takes 16 kHz mono audio, so resample anything else first; phone audio is untested.

How do I try Crele Nimbus?

Install the package and open the playground: press the microphone or play a file, and watch a row per question answer. First use downloads 1.6 GB.

pip install git+https://github.com/Crele-AI/nimbuspython -m crele_nimbus.playground --open

Can I use Crele Nimbus commercially?

Yes. The code and the weights are Apache-2.0. Get the code on GitHub and the weights on Hugging Face.

Questions

Is Crele Nimbus free for commercial use?

Yes. The code and the weights are Apache-2.0.

Does Crele Nimbus need a GPU?

No. It keeps up with live audio on a laptop CPU.

Does my audio go to Crele?

No. Crele Nimbus runs on your own machine, and the audio never leaves it.

How many questions can Crele Nimbus answer at once?

Up to 32 per session. The audio is heard once and every question reads from it, and it keeps up with live audio on a laptop CPU with 32 questions.

What languages does Crele Nimbus understand?

Its questions are in English.

Is there a hosted Crele Nimbus API?

Not yet. If you want Crele Nimbus as a hosted stream, say so on the API waitlist.