Transformers documentation

Parakeet

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.1).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Model ini disumbangkan ke Hugging Face Transformers pada 2025-09-25.

Parakeet

Ikhtisar

Model Parakeet, yang diperkenalkan oleh NVIDIA NeMo, adalah model yang menggabungkan encoder Fast Conformer dengan decoder connectionist temporal classification (CTC), recurrent neural network transducer (RNNT), atau token and duration transducer (TDT) untuk pengenalan ucapan otomatis.

Arsitektur Model

Fast Conformer Encoder: Arsitektur Conformer yang dapat diskalakan secara linear, memproses fitur mel-spectrogram, dan mengurangi panjang sekuens melalui subsampling. Ini adalah versi yang lebih efisien dari Conformer Encoder yang ditemukan di FastSpeech2Conformer (lihat ParakeetEncoder untuk implementasi dan detail encoder).
ParakeetForCTC: Encoder Fast Conformer + decoder CTC
- CTC Decoder: Decoder sederhana namun efektif yang terdiri dari:
  - Proyeksi konvolusi 1D dari ukuran tersembunyi encoder ke ukuran kosakata (untuk kompatibilitas NeMo yang optimal).
  - Perhitungan kerugian CTC untuk pelatihan.
  - Dekoding CTC serakah untuk inferensi.
ParakeetForRNNT: Encoder Fast Conformer + decoder RNN-T (RNN Transducer)
- RNN-T Decoder: Transduser neural standar:
  - Jaringan prediksi LSTM mempertahankan konteks bahasa di antara prediksi token.
  - Jaringan gabungan menggabungkan keluaran encoder dan decoder.
  - Greedy transducer decoding untuk inferensi: emisi blank menggeser frame encoder maju satu, emisi non-blank tetap pada frame yang sama.
ParakeetForTDT: Encoder Fast Conformer + decoder TDT (Token Duration Transducer)
- TDT Decoder: Memprediksi token dan durasinya secara bersamaan, memungkinkan decoding yang efisien:
  - Jaringan prediksi LSTM mempertahankan konteks bahasa di antara prediksi token.
  - Jaringan gabungan menggabungkan keluaran encoder dan decoder.
  - Duration head memprediksi berapa banyak frame yang harus dilewati, memungkinkan inferensi yang cepat.

Implementasi asli dapat ditemukan di NVIDIA NeMo. Checkpoint model dapat ditemukan di bawah organisasi NVIDIA.

Model ini disumbangkan oleh Nithin Rao Koluguri, Eustache Le Bihan, Eric Bezzam, Maksym Lypivskyi, dan Hainan Xu.

Penggunaan

Penggunaan ParakeetForCTC

Pipeline

AutoModel

Penggunaan ParakeetForRNNT

Pipeline

AutoModel

Penandaan Waktu

Parakeet RNN-T mentranskripsi tanpa kapitalisasi atau tanda baca (seperti CTC), dan model ini juga dapat melakukan penandaan waktu token.

from transformers import pipeline

pipe = pipeline("automatic-speech-recognition", model="nvidia/parakeet-rnnt-0.6b", revision="refs/pr/4")
out = pipe("https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/bcn_weather.mp3")
print(out)
# {'text': 'yesterday it was thirty five degrees in barcelona but today the temperature will go down to minus twenty degrees'}

Berbeda dengan TDT (yang memprediksi durasi per token), setiap token RNN-T dipancarkan pada satu frame encoder, sehingga awal dan akhirnya tepat mencakup satu frame.

from datasets import Audio, load_dataset
from transformers import AutoModelForRNNT, AutoProcessor

model_id = "nvidia/parakeet-rnnt-0.6b"
revision = "refs/pr/4"
processor = AutoProcessor.from_pretrained(model_id, revision=revision)
model = AutoModelForRNNT.from_pretrained(model_id, revision=revision, device_map="auto")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el['array'] for el in ds["audio"][:1]]

inputs = processor(speech_samples, sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)
output = model.generate(**inputs, return_dict_in_generate=True)
decoded_output, decoded_timestamps = processor.decode(
    output.sequences,
    durations=output.durations,
    skip_special_tokens=True,
)
print("Transcription:", decoded_output)
print("\nTimestamped tokens:", decoded_timestamps)

"""
Transcription: ['mister quilter is the apostle of the middle classes and we are glad to welcome his gospel']

Timestamped tokens: [[{'token': 'm', 'start': 0.4, 'end': 0.48}, {'token': 'is', 'start': 0.56, 'end': 0.64}, {'token': 'ter', 'start': 0.56, 'end': 0.64}, {'token': ' qu', 'start': 0.72, 'end': 0.8}, {'token': 'il', 'start': 0.96, 'end': 1.04}, {'token': 'ter', 'start': 1.12, 'end': 1.2}, {'token': ' is', 'start': 1.36, 'end': 1.44}, {'token': ' the', 'start': 1.52, 'end': 1.6}, {'token': ' ap', 'start': 1.68, 'end': 1.76}, {'token': 'o', 'start': 1.76, 'end': 1.84}, {'token': 'st', 'start': 1.84, 'end': 1.92}, {'token': 'le', 'start': 2.0, 'end': 2.08}, {'token': ' of', 'start': 2.16, 'end': 2.24}, {'token': ' the', 'start': 2.24, 'end': 2.32}, {'token': ' m', 'start': 2.4, 'end': 2.48}, {'token': 'id', 'start': 2.48, 'end': 2.56}, {'token': 'd', 'start': 2.56, 'end': 2.64}, {'token': 'le', 'start': 2.56, 'end': 2.64}, {'token': ' cl', 'start': 2.8, 'end': 2.88}, {'token': 'ass', 'start': 2.88, 'end': 2.96}, {'token': 'es', 'start': 3.12, 'end': 3.2}, {'token': ' and', 'start': 3.28, 'end': 3.36}, {'token': ' we', 'start': 3.44, 'end': 3.52}, {'token': ' are', 'start': 3.6, 'end': 3.68}, {'token': ' gl', 'start': 3.84, 'end': 3.92}, {'token': 'ad', 'start': 3.92, 'end': 4.0}, {'token': ' to', 'start': 4.08, 'end': 4.16}, {'token': ' we', 'start': 4.24, 'end': 4.32}, {'token': 'l', 'start': 4.32, 'end': 4.4}, {'token': 'c', 'start': 4.4, 'end': 4.48}, {'token': 'ome', 'start': 4.48, 'end': 4.56}, {'token': ' his', 'start': 4.72, 'end': 4.8}, {'token': ' go', 'start': 4.96, 'end': 5.04}, {'token': 's', 'start': 5.04, 'end': 5.12}, {'token': 'pe', 'start': 5.2, 'end': 5.28}, {'token': 'l', 'start': 5.36, 'end': 5.44}]]
"""

Penggunaan ParakeetForTDT

Pipeline

AutoModel

Penandaan Waktu

Membuat Model Berjalan Cepat

Parakeet mendukung kompilasi grafik penuh dengan penangkapan grafik (CUDA graphs pada GPU NVIDIA, dan mekanisme ekuivalen pada akselerator lainnya)! Optimasi ini paling efektif ketika Anda mengetahui panjang audio maksimum yang ingin Anda transkripsi. Ide utamanya adalah menggunakan bentuk input statis untuk menghindari kompilasi ulang. Misalnya, jika Anda tahu audio Anda akan di bawah 30 detik, Anda dapat menggunakan prosesor untuk mengisi semua input hingga 30 detik, menyiapkan fitur input dan mask perhatian yang konsisten. Lihat contoh di bawah ini!

import torch
from datasets import Audio, load_dataset

from transformers import AutoModelForCTC, AutoProcessor


processor = AutoProcessor.from_pretrained("nvidia/parakeet-ctc-1.1b")
model = AutoModelForCTC.from_pretrained("nvidia/parakeet-ctc-1.1b", device_map="auto")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el['array'] for el in ds["audio"][:5]]

# Compile the generate method with fullgraph and graph capture
model.generate = torch.compile(model.generate, fullgraph=True, mode="reduce-overhead")

# let's define processor kwargs to pad to 30 seconds
processor_kwargs = {
    "padding": "max_length",
    "max_length": 30 * processor.feature_extractor.sampling_rate,
}

# Define a timing context using accelerator events
class TimerContext:
    def __init__(self, name="Execution"):
        self.name = name
        self.start_event = None
        self.end_event = None

    def __enter__(self):
        # Use accelerator events for more accurate device timing
        self.start_event = torch.Event(enable_timing=True)
        self.end_event = torch.Event(enable_timing=True)
        self.start_event.record()
        return self

    def __exit__(self, *args):
        self.end_event.record()
        torch.accelerator.synchronize()
        elapsed_time = self.start_event.elapsed_time(self.end_event) / 1000.0
        print(f"{self.name} time: {elapsed_time:.4f} seconds")


inputs = processor(speech_samples[0], **processor_kwargs)
inputs.to(model.device, dtype=model.dtype)
print("\n" + "="*50)
print("First generation - compiling...")
# Generate with the compiled model
with TimerContext("First generation"):
    outputs = model.generate(**inputs)
print(processor.decode(outputs))

inputs = processor(speech_samples[1], **processor_kwargs)
inputs.to(model.device, dtype=model.dtype)
print("\n" + "="*50)
print("Second generation - recording graphs...")
with TimerContext("Second generation"):
    outputs = model.generate(**inputs)
print(processor.decode(outputs))

inputs = processor(speech_samples[2], **processor_kwargs)
inputs.to(model.device, dtype=model.dtype)
print("\n" + "="*50)
print("Third generation - fast !!!")
with TimerContext("Third generation"):
    outputs = model.generate(**inputs)
print(processor.decode(outputs))

inputs = processor(speech_samples[3], **processor_kwargs)
inputs.to(model.device, dtype=model.dtype)
print("\n" + "="*50)
print("Fourth generation - still fast !!!")
with TimerContext("Fourth generation"):
    outputs = model.generate(**inputs)
print(processor.decode(outputs))

Pelatihan CTC

from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor

model_id = "nvidia/parakeet-ctc-1.1b"
NUM_SAMPLES = 5

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, device_map="auto")
model.train()

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el['array'] for el in ds["audio"][:NUM_SAMPLES]]
text_samples = ds["text"][:NUM_SAMPLES]

# passing `text` to the processor will prepare inputs' `labels` key
inputs = processor(audio=speech_samples, text=text_samples, sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)

outputs = model(**inputs)
print("Loss:", outputs.loss.item())
outputs.loss.backward()

Pelatihan RNN-T

Menghitung kerugian RNN-T memerlukan torchaudio (pip install torchaudio).

from datasets import Audio, load_dataset
import torch
from transformers import AutoModelForRNNT, AutoProcessor

model_id = "nvidia/parakeet-rnnt-0.6b"
revision = "refs/pr/4"
NUM_SAMPLES = 4

processor = AutoProcessor.from_pretrained(model_id, revision=revision)
model = AutoModelForRNNT.from_pretrained(model_id, revision=revision,  device_map="auto")
model.train()

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el['array'] for el in ds["audio"][:NUM_SAMPLES]]
text_samples = ds["text"][:NUM_SAMPLES]

# passing `text` to the processor will prepare inputs' `labels` key
inputs = processor(audio=speech_samples, text=text_samples, sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)

outputs = model(**inputs)
print("Loss:", outputs.loss.item())
outputs.loss.backward()

Pelatihan TDT

from datasets import Audio, load_dataset
import torch
from transformers import AutoModelForTDT, AutoProcessor

model_id = "nvidia/parakeet-tdt-0.6b-v3"
NUM_SAMPLES = 4

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForTDT.from_pretrained(model_id, device_map="auto")
model.train()

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el['array'] for el in ds["audio"][:NUM_SAMPLES]]
text_samples = ds["text"][:NUM_SAMPLES]

# passing `text` to the processor will prepare inputs' `labels` key
inputs = processor(audio=speech_samples, text=text_samples, sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)

outputs = model(**inputs)
print("Loss:", outputs.loss.item())
outputs.loss.backward()

Transformers

Parakeet

Ikhtisar

Penggunaan

Penggunaan ParakeetForCTC

Penggunaan ParakeetForRNNT

Penggunaan ParakeetForTDT

Membuat Model Berjalan Cepat

Pelatihan CTC

Pelatihan RNN-T

Pelatihan TDT

ParakeetTokenizer

kelas transformers.ParakeetTokenizer

ParakeetFeatureExtractor

kelas transformers.ParakeetFeatureExtractor

__call__

ParakeetProcessor

kelas transformers.ParakeetProcessor

__call__

decode

ParakeetEncoderConfig

kelas transformers.ParakeetEncoderConfig

ParakeetCTCConfig

kelas transformers.ParakeetCTCConfig

ParakeetRNNTConfig

kelas transformers.ParakeetRNNTConfig

ParakeetTDTConfig

kelas transformers.ParakeetTDTConfig

ParakeetEncoder

kelas transformers.ParakeetEncoder

forward

ParakeetForCTC

kelas transformers.ParakeetForCTC

forward

generate

ParakeetForRNNT

kelas transformers.ParakeetForRNNT

forward

ParakeetForTDT

kelas transformers.ParakeetForTDT

forward

call

call