Nvidia's open Nemotron 3.5 Transcribe combines transcription, translation, and speaker recognition - The Decoder
Nvidia has released an open speech recognition model that combines transcription, translation, and speaker recognition in a single system, claiming top performance in several benchmarks.
Nvidia has released an open speech recognition model that takes text instructions and combines transcription, translation, and speaker recognition capabilities. The company claims Nemotron 3.5 Transcribe leads several benchmarks in these areas. The release also includes a large training dataset and tools for adapting the model to specialized fields.
Unlike basic transcripts that don't identify speakers or translate content, Nemotron 3.5 Transcribe handles all these tasks in a single model. It can also answer questions about a recording's content. Nvidia says the model transcribes 40 languages, translates 31 of them to or from English, and identifies speakers in 13. It processes recordings up to an hour long with up to eight speakers.
Users control the model through simple text instructions. The same meeting recording can become a transcript with speaker labels, an English translation, or a summary of decisions based on the prompt. Users can also provide technical terms like drug names as hints that the model will only use if they appear in the recording.
Top accuracy with efficient processing
The model has 30 billion parameters, but only about 4 billion are active per processing step, allowing it to run efficiently on a single Nvidia GPU. This includes everything from compact DGX Spark to data center hardware. Nvidia claims the model processes 100 hours of audio in under three minutes on an RTX PRO 6000 Blackwell Server Edition at NVFP4 precision.
On Hugging Face's English Open ASR Leaderboard, Transcribe ranks first among open-weight models with an average word error rate of 5.33 percent. In Artificial Analysis' comparison, it scores 2.81 percent, close to Mistral's larger Voxtral Small at 2.77 percent, but Transcribe uses only about a sixth of the active parameters. Several commercial speech services reportedly trail behind.
In Artificial Analysis' comparison, Nemotron 3.5 Transcribe makes barely more errors than Mistral's much larger Voxtral Small while using a fraction of the compute.
Superior speaker recognition
The model shows particular strength in speaker recognition. VoiceArena tested 14 systems on conversations with overlapping speech, and Transcribe took first place. It missed only 8.32 percent of words compared to a median of about 31 percent. On David AI's benchmark measuring speaker attribution across 21 languages, Transcribe posted the lowest error rate, finishing ahead of Nvidia's own dedicated model for the task.
In VoiceArena's comparison, Nemotron 3.5 Transcribe is the best at telling who said what in English conversations where speakers interrupt each other.
Specialized applications with new dataset
Nvidia is releasing the Granary v2 dataset alongside the model, containing about 844,000 hours of speech in 48 languages. Each recording comes in multiple transcription styles, including verbatim with filler words and cleaned versions. Nvidia is also sharing the tools used to build the dataset so developers can adapt them for their own recordings.
The company tested fine-tuning on recordings of Indian doctors and medical students, reducing the error rate from 30.8 to 11.8 percent after adaptation. For new speakers, the error rate dropped from 22.44 to 12.17 percent, a 45.8 percent relative improvement that outperformed an open medical speech model. However, the specialized model showed slightly higher error rates on general English speech.
The base model has limitations, with features varying by language. The audio question-answering feature was trained in English only, and the model is designed for recorded rather than live audio. Nvidia recommends users spot-check names, speaker changes, and answers. The model and dataset are available on Hugging Face, along with a demo and fine-tuning guides.
Related
