EG2 -> Qwen-Image 2.1 conditioning adapter, v3 (permissive data, research release)

A 107M-parameter adapter that lets EmbeddingGemma 2 (EG2, 0.74B, Apache-2.0) replace the 8B Qwen3-VL text encoder of Qwen-Image 2.1. Because EG2 embeds text, sounds and video frames through one language model, the same adapter lets Qwen-Image's edit path take sounds and short videos as references (audio1, video1, up to nine each) where Qwen-Image takes image1. Every training item is CC0 or CC BY; the generated attribution list ships here.

sounds as the only prompt: acoustic guitar, bowed string, hi-hat, bird

Try it in the browser (free ZeroGPU): https://huggingface.co/spaces/oakoliver/eg2-qwen-image-sounds-videos (upload a sound or a short video, pick Turbo or Base with your own number of steps).

Release terms (read first)

  • These adapter weights are released for research and evaluation only, under the Qwen Research License Agreement (LICENSE). The adapter is distilled from Qwen-Image 2.1's text encoder and trained through its diffusion transformer, so we treat it as a derivative work of those Materials. Commercial use needs Alibaba's separate commercial license for Qwen-Image 2.1 (model-business@notice.qwencloud.com). See NOTICE.
  • EmbeddingGemma 2 is Apache-2.0 and is downloaded from Google separately; nothing from EG2 is redistributed here.
  • Training data: FSD50K dev clips (CC0 / CC BY 3.0, Fonseca et al. 2022), Wikimedia Commons videos (CC0 / CC BY 3.0 / CC BY 4.0, licenses as declared by the uploaders), COCO captions (CC BY 4.0), Localized Narratives (CC BY 4.0), DOCCI (CC BY 4.0), DiffusionDB prompts (CC0). Per-item attribution: v3_attribution.md (19.6k lines). Evaluation-only sets, never trained on: ESC-50 (CC BY-NC), MSR-VTT (research), FSD50K eval clips (CC0 / CC BY, v3_audio_eval_attribution.md).
  • Our previous adapters (trained on ShareGPT4V, ESC-50 and MSR-VTT) stay private for data-license reasons.

Files

file what
adapter.safetensors the adapter (sha256 ab4dbd89โ€ฆ8e1b7), export of stage 2 + phase D step 6500
config.json architecture: 8 pre-LN layers, 1024 wide, 16 heads, FFN 4096, 768 -> 4096, 3 constant head tokens, 5 learned tail queries, EG2 prefix
LICENSE, NOTICE Qwen Research License Agreement and the required notice
v3_attribution.md, v3_audio_eval_attribution.md generated attribution lists
bench/ rig text benchmarks (four checkpoints), FSD50K eval audio bench, Mac reference bench
examples/ sound-only and video-only generations from held-out clips

How it was trained

  1. Stage 1, text distillation (20k steps x 16 captions, one RTX 3090): EG2 per-token states -> adapter -> the teacher encoder's last-layer hidden states for the same caption (right padding, 14 system tokens dropped), MSE in per-channel standardized units + 0.1 cosine. 1.95M captions (COCO 590k, Localized Narratives 687k, DOCCI 77k = 9.6k x 8, DiffusionDB 600k). Final cosine 0.902.
  2. Stage 2 + phase D, 6,500 steps: velocity distillation through the frozen DiT over 14 resolution buckets (512ยฒ to 2048 x 1152) plus reference alignment: the student sees EG2 text with <|audio|> / <|video|> slots filled by a clip (25 soft tokens per second of audio, up to 6 frames of video), the teacher sees the caption that names it; tail MSE 0.5, pooled MSE 0.25, InfoNCE 0.1, with the stage-1 text loss kept in the mix so text fidelity does not drift. 14,977 FSD50K sounds (19.6 h, all 200 classes), 4,473 Commons clips captioned by Qwen3-VL-8B-Instruct.

Results

Text prompts, adapter vs the Qwen3-VL teacher (rig, 40 steps, 62 prompts, paired per prompt, bootstrap 95 % CI):

metric adapter teacher gap
SigLIP2 x100 15.90 17.35 -1.45 [-2.36, -0.68]
CLIP-L 26.57 28.08 -1.51 [-2.33, -0.78]
PickScore 22.84 23.35 -0.52 [-0.84, -0.23]
GenEval-style pass (OWLv2, 36 prompts) 69.4 % 77.8 % not significant
OCR word accuracy, rendered text (6 prompts) 41.7 % 83.3 % not significant (n = 6)

Spatial relations are the one clear loss (50 % vs 100 % on six prompts). Our previous best adapter on the same suite: SigLIP2 -1.71, GenEval 69.4 %, OCR 16.7 %.

References, zero-shot retrieval over held-out clips the adapter never trained on:

test result chance
MSR-VTT video1, 100 clips, Mac engine 512ยฒ 8 steps, SigLIP2 R@1 / R@5 / R@10 67 / 95 / 98 % 1 / 5 / 10 %
same, CLIP-L judge 61 / 90 / 94 %
adapter's own text ceiling on those captions 50 / 78 / 83 %
ESC-50 audio1, 100 clips, 50-way CLIP top-1 / top-5 24 / 48 % 2 / 10 %
FSD50K eval audio1, 100 clips, 100-way CLIP top-1 / top-5 / median rank 14 / 28 % / 19.5 1 / 5 % / 50.5
adapter text ceiling on the FSD50K classes (teacher 90 / 99) 81 / 91 %

Sounds with an obvious visual (instruments, birds, clapping, fireworks, laughter, singing, cough) are recognised at or near 100 %; abstract or texture sounds (wind, zipper, typewriter, running tap, writing) at 0 %.

laughter, cough, clapping, fireworks near misses: church bell -> cello, trumpet -> sax, zipper -> dog, train -> truck videos as the only prompt

How to run

With the qwen-image-rs engine (Rust, Metal, EG2 backend; EG2 is read from the Hugging Face cache):

QWEN_VAE_F32_CONV=1 qwen-image generate --text-backend eg2 --eg2-adapter ./ \
  --ref-audio dog.wav --ref-video 'clip.mkv#t=12,18' --ref-weight-mode repeat \
  --steps 40 --scheduler official --prompt "a storybook illustration of audio1 in the scene of video1"

path#t=start,end selects a time range; #ch=split splits stereo; mp3 / flac / ogg / m4a and video soundtracks decode through ffmpeg. qwen-image batch and serve keep the weights resident. Text stage 8 to 33 ms vs 81 to 148 ms for the Qwen3-VL encoder; peak memory 21.7 vs 32.6 GB at 1024ยฒ (Apple M-series).

Known limits

Image references are semantic only (no identity transfer; use Qwen-Image's own latent edit path for that). Rendered text is below the teacher. The audio tower runs on the host in f32 (about 2 s per 2 s clip); the vision tower about 9.6 s per 4-frame video. Spatial relations between two objects are the weakest text category.

Credits

Qwen-Image 2.1 and its text encoder: Alibaba / Hangzhou Tongyi Laboratory. EmbeddingGemma 2: Google. FSD50K: Fonseca et al., IEEE/ACM TASLP 2022. Wikimedia Commons contributors, COCO Consortium, Google (Localized Narratives, DOCCI), Polo Club (DiffusionDB): see the attribution files. Write-up: https://engineering.oakoliver.com/articles/sounds-and-videos-as-prompts-embeddinggemma-2-adapter-qwen-image-iris

Downloads last month
59
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for oakoliver/eg2-qwen-image-2.1-adapter

Finetuned
(100)
this model

Space using oakoliver/eg2-qwen-image-2.1-adapter 1