model.generate() crashes with decoder_attention_mask=None on transformers 4.57
Hit this while evaluating the model on FLEURS the recommended snippet from the model card crashes immediately:
AttributeError: 'NoneType' object has no attribute 'new_ones'
Trace lands inside _update_model_kwargs_for_generation trying to extend the decoder mask after each token. Reproduces on a single audio sample, no batching weirdness.
Repro
import torch, soundfile as sf
from huggingface_hub import hf_hub_download
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
MID = "CohereLabs/cohere-transcribe-03-2026"
processor = AutoProcessor.from_pretrained(MID, trust_remote_code=True)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
MID, trust_remote_code=True, device_map="auto", torch_dtype=torch.bfloat16
).eval()
wav = hf_hub_download(MID, "demo/voxpopuli_test_en_demo.wav")
audio, _ = sf.read(wav)
inputs = processor([audio], sampling_rate=16000, return_tensors="pt", language="en")
inputs = inputs.to(model.device, dtype=model.dtype)
model.generate(**inputs, max_new_tokens=64) # boom
Env: transformers==4.57.6, torch==2.8.0+cu128, torchaudio==2.8.0+cu128. (Same with transformers==4.55.)
What's going wrong
modeling_cohere_asr.py line 906 sets decoder_attention_mask into generation_kwargs unconditionally but decoder_attention_mask only gets built when decoder_input_ids is also passed (lines 897-899). For a normal generate call neither is passed, so a None ends up in model_kwargs, and transformers' newer _update_model_kwargs_for_generation then calls .new_ones(...) on it.
# line 896-906, modeling_cohere_asr.py
decoder_attention_mask = kwargs.pop("decoder_attention_mask", None)
if decoder_input_ids is not None and decoder_attention_mask is None:
decoder_attention_mask = torch.ones_like(
decoder_input_ids, dtype=torch.long, device=decoder_input_ids.device
)
generation_kwargs = dict(kwargs)
generation_kwargs["input_features"] = input_features
generation_kwargs["length"] = length
generation_kwargs["decoder_input_ids"] = decoder_input_ids
generation_kwargs["decoder_attention_mask"] = decoder_attention_mask # <-- None gets through
Suggested fix (1 line)
- generation_kwargs["decoder_attention_mask"] = decoder_attention_mask
+ if decoder_attention_mask is not None:
+ generation_kwargs["decoder_attention_mask"] = decoder_attention_mask
Workaround until then
Pre-populate an empty mask before calling generate:
inputs["decoder_attention_mask"] = torch.zeros(
inputs["input_features"].shape[0], 0, dtype=torch.long, device=model.device
)
model.generate(**inputs, max_new_tokens=64)
Verified across ~12k samples per language on FLEURS-102 with this workaround in place no other regressions noticed.
Created PR #35 for the 1 liner fix
Hi - apologies for the confusion. I think you are attempting to mix two incompatible usage patterns in the repro above. Unless I'm missing something, I don't think your repro follows the recommended pattern of the current or previous snippets.
The two usage patterns are:
- recommended: Use
transformers>=5.4.0without trust_remote_code=True. Follow quick start pattern in the docs. - deprecated: If you really need
transfomers==4.57you can look back at the old docs in the HF repo. This version ran with trust_remote_code=True. But in this case it was required to use thetranscribe()API rather thanmodel.generate()in your repro above
To give a bit more context, the native support for cohere-transcribe landed in transformers>=5.4.0 and this is significantly faster due to the added rust tokenizer. We recently removed all trust_remote_code=True snippets from the model card to avoid confusion going forward. This means we don't recommend using transformers<5.4.0 if you can avoid it. However, if you have a requirement for transformers==4.57 and find an issue in the old docs for trust_remote_code=True when respecting the required dependency ranges we'll be happy to look into it 🤗 !
Thanks for the context that clears it up. You're right, I was mixing patterns; didn't notice transcribe() in the modeling file until your reply. We're pinned to transformers==4.57 for now due to other constraints in the same eval pipeline (Qwen3-ASR has tight transformers pins), so we were trying the custom-modeling path but via the wrong entry point.
The fix isn't needed if nobody's supposed to be calling generate() via the custom code anyway. Will switch to transcribe() if we stay on 4.57, or the native transformers path once we can upgrade.
Great - glad you are unblocked (and thanks for the update!)