Crele Nimbus A real-time audio decision model
Crele Nimbus answers yes/no questions about live audio in plain English, every 40 ms, about a tenth of a second after the moment. It also knows whether the user or someone else is speaking.
Is there an open-source Jev model for audio?
Crele Nimbus answers Jev-style yes/no questions about live audio as it streams, with open weights under Apache-2.0. Jev reads text and does not take audio, so with Jev, sound has to become a transcript first.
# POST https://api.typesafe.ai/v1/systemone{ "model": "jev-latest", "state": "Caller: sorry, can you say that again?", "questions": { "tv": { "type": "noul", "instructions": "Is a TV playing in the room?", "criteria": { "true": "A TV, radio or video is playing", "false": "Only people in the room" } } }}# the same question, asked of the audio itselfsession.ask( "tv", "Is a TV playing in the room?", yes_means="A TV, radio or video is playing", no_means="Only people in the room",) for update in session.push(mic_16k): update.answers["tv"].answer # yes, no or cant_tell for event in update.events: # started, stopped ...| Property | Jev (TypeSafe) | Crele Nimbus |
|---|---|---|
| Reads | Text in state; no audio or video | Live audio, 16 kHz, pushed in any chunk size |
| Question types | noul (yes/no), choice, score | Yes/no |
| Your meaning | criteria, true and false for a noul | yes_means, no_means |
| Answers | A noul is a number from 0 to 1, per call | Yes, no or can't tell with the probability, every 40 ms, plus started and stopped events |
| Who is speaking | Whatever the transcript says | Built in: the user, from a few seconds of their voice, and anyone else |
| Runs | Hosted, by TypeSafe and its providers | On your own machine, real time on a laptop CPU, Apache-2.0 |
How does Crele Nimbus tell the user from everyone else?
Crele Nimbus does personal voice activity detection: from a few seconds of the user's voice, it says when the user starts and stops talking, apart from anyone else in the room or on a TV.
Barge-in from the user only
Stop talking when the user interrupts, and keep going through the TV.
When the user stops talking
Reply when the user's voice stops, even while others keep talking.
Sounds around the call
Music, laughter, a siren: ask about them in your own words.
Someone joins
Notice another person or a TV voice, and adapt.
How fast is Crele Nimbus?
Crele Nimbus says yes about 0.2 s after a sound starts, and runs in real time on a laptop CPU.
- Every 40 ms
- An answer every 40 ms, delivered every 80 ms: 80 to 120 ms of audio after the moment, plus compute.
- On a laptop CPU
- Keeps up with live audio on an Apple M4 Pro CPU with 8 and with 32 questions.
- Speech
- A median of 0.21 s from the start of speech to a yes, with 97% of 181 speech onsets caught.
- In your own words
- Reworded questions it never trained on scored 0.892 against 0.911 for its own wording (mean AUROC, 40 sounds, the same 500 clips).
- In a silent room
- The highest resting answer in a silent room was 0.03 across twelve sound questions.
- Size
- About 97M parameters on the audio, plus a 0.6B text encoder used once per question.
How do I run Crele Nimbus?
Install the package, stream audio into a session and read the answers, or try it on your microphone in the playground.
Step 1
Install the package
Python 3.10 or newer, PyTorch 2.4 or newer. First use downloads 1.6 GB: the weights and the text encoder that reads the questions.
Terminalpip install git+https://github.com/Crele-AI/nimbusStep 2
Ask questions and stream audio
Push 16 kHz mono audio in any chunk size. A few seconds of the user's voice turn on the user signal.
Pythonfrom crele_nimbus import Nimbus model = Nimbus.from_pretrained("crele/nimbus-v1")session = model.session()session.ask("music", "Is music playing?")session.enroll(user_voice_16k) # a few seconds of the user for update in session.push(pcm_16k): # float32 mono, any length for event in update.events: if event.question == "user" and event.kind == "started": agent.stop() # the user interruptedStep 3
Try it on your microphone
One page on your own machine: press the microphone or play a file, and watch a row per question answer.
Terminalpython -m crele_nimbus.playground --open
What are Crele Nimbus's limits?
Crele Nimbus takes English questions, one user voice per session, and 16 kHz mono audio.
- English questions
- Questions are asked in English.
- One user voice
- One reference voice per session. The user signal answers can't tell until it has heard a few seconds of it.
- The first second
- Every question answers can't tell for the first second of a stream, while the model gathers context.
- 16 kHz mono
- Resample anything else to 16 kHz first. Phone audio is untested.
- First use
- Downloads 1.6 GB into the Hugging Face cache: the weights and the text encoder that reads the questions.
- One thread
- A session is called from one thread; the playground shows the pattern for async code.
Questions
- Is there a local Jev model for audio?
- TypeSafe offers Jev as a hosted API, and Jev reads text rather than audio. Crele Nimbus answers Jev-style yes/no questions about live audio on your own machine, under Apache-2.0.
- Can Jev process audio?
- No. TypeSafe's documentation says Jev does not accept audio or video and asks you to turn them into text first; Crele Nimbus answers yes/no questions about the audio itself.
- What is personal voice activity detection?
- Voice activity detection that follows one known person: it says when that person is talking, not when anyone is. Crele Nimbus does it live from a few seconds of the user's voice, beside a second signal for everyone else.
- How is Crele Nimbus different from voice activity detection?
- A voice activity detector answers one question: is anyone speaking? Crele Nimbus answers whether the user is speaking, whether someone else is, and any yes/no question you ask about the other sounds.
- How do I make a voice agent ignore the TV?
- Stop the agent on the user's voice, not on any voice. With Crele Nimbus, act on the user signal's started event; a TV or another person shows up on the other signal instead.
- Can Crele Nimbus run on a CPU?
- Yes. It keeps up with live audio on an Apple M4 Pro laptop CPU with 8 and with 32 questions.
- How do I detect if music is playing in real time?
- Ask Crele Nimbus "Is music playing?" and read its answer every 40 ms, or act on the started and stopped events. Any other sound works the same way, asked in plain English.
- Is Crele Nimbus free for commercial use?
- Yes. The code and the weights are Apache-2.0.
- How many questions can Crele Nimbus answer at once?
- Up to 32 per session. The audio is heard once and every question reads from it, and it keeps up with live audio on a laptop CPU with 32 questions.
- Does my audio go to Crele?
- No. Crele Nimbus runs on your own machine, and the audio never leaves it.
- What languages does Crele Nimbus understand?
- Its questions are in English.
- What is Crele Nimbus built on?
- A streaming audio encoder with the structure of Dasheng-Base, the CAM++ speaker model for the two voice signals, and Qwen3-Embedding-0.6B, which reads each question once. It was trained on AudioSet and AudioSet-Strong, LibriSpeech, MUSAN and FSD50K.
- Is there a hosted Crele Nimbus API?
- Not yet. If you want Crele Nimbus as a hosted stream, say so on the API waitlist.
Why we built it: Introducing Crele Nimbus
Want it hosted? Join the API waitlist