EG2 -> Qwen-Image 2.1 conditioning adapter, v3 (permissive data, research release)
A 107M-parameter adapter that lets EmbeddingGemma 2 (EG2, 0.74B, Apache-2.0) replace the 8B Qwen3-VL text encoder
of Qwen-Image 2.1. Because EG2 embeds text, sounds and video frames through one language model, the same adapter
lets Qwen-Image's edit path take sounds and short videos as references (audio1, video1, up to nine each) where
Qwen-Image takes image1. Every training item is CC0 or CC BY; the generated attribution list ships here.
Try it in the browser (free ZeroGPU): https://huggingface.co/spaces/oakoliver/eg2-qwen-image-sounds-videos (upload a sound or a short video, pick Turbo or Base with your own number of steps).
Release terms (read first)
- These adapter weights are released for research and evaluation only, under the Qwen Research License Agreement
(
LICENSE). The adapter is distilled from Qwen-Image 2.1's text encoder and trained through its diffusion transformer, so we treat it as a derivative work of those Materials. Commercial use needs Alibaba's separate commercial license for Qwen-Image 2.1 (model-business@notice.qwencloud.com). SeeNOTICE. - EmbeddingGemma 2 is Apache-2.0 and is downloaded from Google separately; nothing from EG2 is redistributed here.
- Training data: FSD50K dev clips (CC0 / CC BY 3.0, Fonseca et al. 2022), Wikimedia Commons videos (CC0 / CC BY 3.0 /
CC BY 4.0, licenses as declared by the uploaders), COCO captions (CC BY 4.0), Localized Narratives (CC BY 4.0),
DOCCI (CC BY 4.0), DiffusionDB prompts (CC0). Per-item attribution:
v3_attribution.md(19.6k lines). Evaluation-only sets, never trained on: ESC-50 (CC BY-NC), MSR-VTT (research), FSD50K eval clips (CC0 / CC BY,v3_audio_eval_attribution.md). - Our previous adapters (trained on ShareGPT4V, ESC-50 and MSR-VTT) stay private for data-license reasons.
Files
| file | what |
|---|---|
adapter.safetensors |
the adapter (sha256 ab4dbd89โฆ8e1b7), export of stage 2 + phase D step 6500 |
config.json |
architecture: 8 pre-LN layers, 1024 wide, 16 heads, FFN 4096, 768 -> 4096, 3 constant head tokens, 5 learned tail queries, EG2 prefix |
LICENSE, NOTICE |
Qwen Research License Agreement and the required notice |
v3_attribution.md, v3_audio_eval_attribution.md |
generated attribution lists |
bench/ |
rig text benchmarks (four checkpoints), FSD50K eval audio bench, Mac reference bench |
examples/ |
sound-only and video-only generations from held-out clips |
How it was trained
- Stage 1, text distillation (20k steps x 16 captions, one RTX 3090): EG2 per-token states -> adapter -> the teacher encoder's last-layer hidden states for the same caption (right padding, 14 system tokens dropped), MSE in per-channel standardized units + 0.1 cosine. 1.95M captions (COCO 590k, Localized Narratives 687k, DOCCI 77k = 9.6k x 8, DiffusionDB 600k). Final cosine 0.902.
- Stage 2 + phase D, 6,500 steps: velocity distillation through the frozen DiT over 14 resolution buckets
(512ยฒ to 2048 x 1152) plus reference alignment: the student sees EG2 text with
<|audio|>/<|video|>slots filled by a clip (25 soft tokens per second of audio, up to 6 frames of video), the teacher sees the caption that names it; tail MSE 0.5, pooled MSE 0.25, InfoNCE 0.1, with the stage-1 text loss kept in the mix so text fidelity does not drift. 14,977 FSD50K sounds (19.6 h, all 200 classes), 4,473 Commons clips captioned by Qwen3-VL-8B-Instruct.
Results
Text prompts, adapter vs the Qwen3-VL teacher (rig, 40 steps, 62 prompts, paired per prompt, bootstrap 95 % CI):
| metric | adapter | teacher | gap |
|---|---|---|---|
| SigLIP2 x100 | 15.90 | 17.35 | -1.45 [-2.36, -0.68] |
| CLIP-L | 26.57 | 28.08 | -1.51 [-2.33, -0.78] |
| PickScore | 22.84 | 23.35 | -0.52 [-0.84, -0.23] |
| GenEval-style pass (OWLv2, 36 prompts) | 69.4 % | 77.8 % | not significant |
| OCR word accuracy, rendered text (6 prompts) | 41.7 % | 83.3 % | not significant (n = 6) |
Spatial relations are the one clear loss (50 % vs 100 % on six prompts). Our previous best adapter on the same suite: SigLIP2 -1.71, GenEval 69.4 %, OCR 16.7 %.
References, zero-shot retrieval over held-out clips the adapter never trained on:
| test | result | chance |
|---|---|---|
MSR-VTT video1, 100 clips, Mac engine 512ยฒ 8 steps, SigLIP2 R@1 / R@5 / R@10 |
67 / 95 / 98 % | 1 / 5 / 10 % |
| same, CLIP-L judge | 61 / 90 / 94 % | |
| adapter's own text ceiling on those captions | 50 / 78 / 83 % | |
ESC-50 audio1, 100 clips, 50-way CLIP top-1 / top-5 |
24 / 48 % | 2 / 10 % |
FSD50K eval audio1, 100 clips, 100-way CLIP top-1 / top-5 / median rank |
14 / 28 % / 19.5 | 1 / 5 % / 50.5 |
| adapter text ceiling on the FSD50K classes (teacher 90 / 99) | 81 / 91 % |
Sounds with an obvious visual (instruments, birds, clapping, fireworks, laughter, singing, cough) are recognised at or near 100 %; abstract or texture sounds (wind, zipper, typewriter, running tap, writing) at 0 %.
How to run
With the qwen-image-rs engine (Rust, Metal, EG2 backend; EG2 is read from the Hugging Face cache):
QWEN_VAE_F32_CONV=1 qwen-image generate --text-backend eg2 --eg2-adapter ./ \
--ref-audio dog.wav --ref-video 'clip.mkv#t=12,18' --ref-weight-mode repeat \
--steps 40 --scheduler official --prompt "a storybook illustration of audio1 in the scene of video1"
path#t=start,end selects a time range; #ch=split splits stereo; mp3 / flac / ogg / m4a and video soundtracks decode
through ffmpeg. qwen-image batch and serve keep the weights resident. Text stage 8 to 33 ms vs 81 to 148 ms for the
Qwen3-VL encoder; peak memory 21.7 vs 32.6 GB at 1024ยฒ (Apple M-series).
Known limits
Image references are semantic only (no identity transfer; use Qwen-Image's own latent edit path for that). Rendered text is below the teacher. The audio tower runs on the host in f32 (about 2 s per 2 s clip); the vision tower about 9.6 s per 4-frame video. Spatial relations between two objects are the weakest text category.
Credits
Qwen-Image 2.1 and its text encoder: Alibaba / Hangzhou Tongyi Laboratory. EmbeddingGemma 2: Google. FSD50K: Fonseca et al., IEEE/ACM TASLP 2022. Wikimedia Commons contributors, COCO Consortium, Google (Localized Narratives, DOCCI), Polo Club (DiffusionDB): see the attribution files. Write-up: https://engineering.oakoliver.com/articles/sounds-and-videos-as-prompts-embeddinggemma-2-adapter-qwen-image-iris
- Downloads last month
- 59
Model tree for oakoliver/eg2-qwen-image-2.1-adapter
Base model
Qwen/Qwen-Image-2.1


