aufklarer/Nemotron-3.5-ASR-Streaming-0.6B-MLX-5bit

Nemotron-3.5 ASR Streaming 0.6B — MLX INT5

Cache-aware multilingual Conformer + RNN-T speech recognition from NVIDIA, converted to native MLX for streaming inference on Apple silicon. This bundle uses affine group-64 5-bit linear weights, BF16 recurrent state, and unquantized convolution, embedding, LSTM, and language-prompt layers.

Model

Parameters	0.6B
Architecture	Cache-aware FastConformer + prompt-conditioned RNN-T
Sample rate	16 kHz mono
Streaming chunk	320 ms
Encoder frame rate	80 ms
Attention left context	56 frames (4.48 s)
Vocabulary	13,087 SentencePiece tokens + blank
Quantization	MLX affine group-64 5-bit linear weights
On-disk size	538.6 MB

Files

File	Size	Description
model.safetensors	538.1 MB	MLX model weights
config.json	1.7 KB	Strict runtime and streaming geometry
vocab.json	100.9 KB	Ordered SentencePiece vocabulary
languages.json	2.0 KB	Wrapped language-to-prompt-slot mapping
lang2slot.json	1.7 KB	Flat compatibility language mapping
tokenizer.model	397.0 KB	Original SentencePiece model
speech_models_export.json	1.3 KB	Pinned source and export artifact hashes
checksums.json	generated	SHA-256 for every published file

Streaming accuracy

FLEURS test, 50 samples per language, cache-aware 320 ms streaming. English uses Whisper's EnglishTextNormalizer; German, French, and Arabic use BasicTextNormalizer; Hindi and Japanese use BasicTextNormalizer(split_letters=True).

Language	WER %	CER %	RTF
en-US	8.64	3.77	0.0447
de-DE	11.15	6.25	0.0436
fr-FR	13.10	5.18	0.0442
ar	13.66	3.94	0.0436
hi-IN	4.77	3.85	0.0441
ja-JP	17.86	12.11	0.0422

Mean WER across these six slices is 11.53% and mean CER is 5.85%.

Streaming performance

M5 Pro with 48 GB unified memory, one fresh Python process, 63.7 seconds of concatenated FLEURS English audio, 199 × 320 ms chunks:

metric	value
Real-time factor	0.0467
p50 chunk latency	13.8 ms
p95 chunk latency	15.9 ms
p99 chunk latency	16.8 ms
RSS after lazy weight load	196 MB
Peak RSS during streaming	800 MB

The post-load value reflects MLX's lazy memory mapping; peak RSS is the useful resident-memory figure after inference touches the weights.

Swift usage

The native runtime is in NemotronStreamingASR.

import NemotronStreamingASR

let model = try await NemotronStreamingASRMLXModel.fromPretrained(
    variant: .int5
)
let session = try model.createSession(language: "en-US")

for pcm320ms in audioChunks {
    for partial in try session.pushAudio(pcm320ms) {
        print(partial.text)
    }
}

for final in try session.finalize() {
    print(final.text)
}

Sessions retain the mel, attention, convolution, and RNN-T predictor caches. Model calls are serialized so multiple source-local sessions can safely share one loaded weight set.

Python reference runtime

The reproducible exporter, cache-aware runtime, tests, and benchmarks live in speech-models/models/nemotron-asr-streaming-multilingual/export.

hf download aufklarer/Nemotron-3.5-ASR-Streaming-0.6B-MLX-5bit --local-dir ./nemotron-mlx

python streaming.py \
  --bundle ./nemotron-mlx \
  --wav recording.wav \
  --lang en-US \
  --chunk-ms 320 \
  --print-partials

--print-partials prints cumulative text whenever a chunk adds visible tokens. Both reference runtimes remove angle-bracket language markers such as <en-US> from user-facing text.

Choosing a precision

INT5 minimizes resident memory; INT8 preserves more quantization headroom. Benchmark both on domain audio before deployment.

Limitations

• This is ASR only. It does not perform speaker diarization or speaker identification.
• Language conditioning is explicit. Pass a supported BCP-47 language tag or use the bundle's auto slot.
• Partial text is provisional and may end mid-word. Applications should replace the previous partial until the caller finalizes the utterance.
• The native MLX path does not currently implement the Core ML runtime's optional word-boosting layer.
• The runtime is intended for Apple silicon and requires the MLX Metal shader library to be packaged with the executable.

Provenance

Upstream repository	nvidia/nemotron-3.5-asr-streaming-0.6b
Upstream revision	f3d333391852ba876df169dcc9ba902d25b6ab0b
Upstream .nemo SHA-256	210214ed94039bf6bfbb9a047c7fa289628db75b103e2bf6381fa78285436a74
Export manifest	speech_models_export.json
Artifact checksums	checksums.json

License

Distributed under OpenMDW 1.1, matching the upstream NVIDIA model. Review the upstream model card and license before deployment.

Links

• speech-swift
• speech-models
• Soniqo documentation
• NVIDIA source model