Listen-on-Demand LoRA adapters for Qwen3-Omni-30B-A3B-Instruct

Three LoRA adapters that teach Qwen3-Omni-30B-A3B-Instruct when to inspect the audio of a video through a text-protocol tool and what to do with the tool's output. They come from the Listen-on-Demand project (sequel to When Vision Speaks for Sound, arXiv:2605.16403); the full project — data, code, evaluation rows, documents — is in Rakancorle1/listen-on-demand. Project status: paused 2026-09-18.

folder	trained	training clips	rows	steps	recipe (details: archive `docs/STATUS.md` §4c / §4e / §4h)
`v1/`	2026-09-05	160 VGG-SS (grid_v2 train)	3,344	210	constructed `CALL_AUDIO_TOOL` lines + blind Claude Opus 5 round-2 answers, Sonnet 5 filter
`v1_1/`	2026-09-06	160	5,396	338	round-2 targets = v1's own K=4 samples (70 % kept) + 703 Opus rows with label semantics; tool wording fix "No speech transcribed."
`v1_2/`	2026-09-10	160 + 774 (grid_v3 train, ≤ 8 clips/class, 219 classes)	32,422	2,028	v1.1 recipe with v1.1 as the sampler (82 % kept), shared `NO_FALSE_ALARM` filter; true per-token CE 0.92 → 0.24

Each folder holds final/ (the adapter used in every reported number), the two epoch checkpoints (checkpoint-*/, with optimizer/scheduler/RNG state for resuming), log_history.json and train_manifest.json (exact row composition, args, world size). v1_1/final is the reference adapter by the PI's ruling of 2026-09-08; v1_2/final is better on 8 of the 12 headline cells and whether it replaces v1.1 is an open decision (archive docs/STATUS.md §5).

Training configuration (all three)

LoRA r = 16, α = 32, dropout 0.05 on the thinker text decoder's attention projections only — ^model\.layers\.\d+\.self_attn\.(q|k|v|o)_proj$, 192 modules, 13.4 M trainable parameters; audio tower, vision encoder and the MoE experts untouched. Loss on assistant tokens through <|im_end|>. lr 1e-4 cosine, 2 epochs, per-device batch 1 (variable-shape multimodal inputs) × grad-accum 4 × 8 H200 (one full model copy per GPU, data-parallel). PEFT 0.18.1, trainer sft/train_lora.py in the archive. Prompts are byte-identical to the evaluation runner's.

adapter_config.json has an empty base_model_name_or_path because training loaded the base model from a local path — pass the base model explicitly when loading.

Loading

from transformers import Qwen3OmniMoeForConditionalGeneration, Qwen3OmniMoeProcessor
from peft import PeftModel

base = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
model = Qwen3OmniMoeForConditionalGeneration.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
processor = Qwen3OmniMoeProcessor.from_pretrained(base)
# the adapter targets the thinker sub-module
model.thinker = PeftModel.from_pretrained(model.thinker, "Rakancorle1/listen-on-demand-adapters", subfolder="v1_2/final")
model.thinker = model.thinker.merge_and_unload()

The project's own runner does exactly this and also implements the protocol: python sft/run_agentic_baseline.py --adapter <path to v1_2/final> ... (archive repo). The protocol is a text protocol, not API function calling: round 1 = video + audio + TOOL_INSTRUCTION + question → the model either answers or replies with the single line CALL_AUDIO_TOOL; round 2 = the tool's evidence string (RMS silence gate → silero-VAD → whisper-large-v3 transcript → AST top-5 tags; "No speech transcribed." when the gated transcript is empty) + "Now answer the question". Train and eval must use the same evidence wording (--legacy-evidence reproduces the v1-era string).

Results (identical runner, Claude Opus 5 grader; accuracy %, baseline → v1.1 → v1.2)

original = no false alarm on untouched audio (hedge_ok criterion = the training filter's question), mute = says silent, swap = says mismatch. Oops/FailArmy = THUD-Eval, a different source from the training clips; n = 142 per cell.

set	type	original	mute	swap
Oops / FailArmy	explicit	98.6 → 75.4 → 75.4	81.7 → 100 → 100	4.2 → 33.1 → 33.8
Oops / FailArmy	implicit	97.9 → 85.1 → 78.9	7.7 → 72.5 → 88.7	0.7 → 33.1 → 27.5
grid_v2 val (20 same-source clips)	explicit	90 → 85 → 85	85 → 100 → 100	45 → 85 → 85
grid_v2 val	implicit	100 → 90 → 100	5 → 85 → 100	0 → 65 → 80
grid_v3 val (97 same-source clips)	explicit	90.8 → 86.2 → 87.4	88.7 → 100 → 100	46.4 → 84.5 → 87.6
grid_v3 val	implicit	100 → 93.8 → 94.8	0.0 → 90.7 → 93.8	3.1 → 74.0 → 80.2

Tool-call rate on questions that do not mention sound (implicit): 14–38 % at baseline → 100 % for every adapter, while visual-only questions stay at 0–3 % — the decision became a question-type decision, not "always call".
Guard cell (vision_dominant answers must not raise an audio conflict): 99.8–100 % for every adapter.
Cost: false alarms on untouched Oops audio 1.2 % → 11.8 / 11.7 / 12.0 % (v1 / v1.1 / v1.2); ~78 % are faithful reports of tool output that lacked the expected sound, the rest genuine over-suspicion. This column did not move in three rounds.
Oops swap is a corpus ceiling (within-compilation swaps; every adapter 27–34 %); class-disjoint same-source swaps reach 80 %.

Licence

Adapter weights: Apache 2.0 (same as the base model). The training text was written by Claude Opus 5 / Sonnet 5 and by the model itself over VGG-SS clips; see the archive repo for provenance.