YuE2-instrumental-cot-full-loras

AR-branch LoRA for YuE2-3B that makes the model write instrumental music with a section plan. Rank 64 on self_attn.{q,k,v,o}_proj + mlp.{gate,up,down}_proj of all 28 layers, trained on ~2.7k instrumental tracks paired with chord-annotated ABC scores from SheetSage2, regularized 50/50 with YuE2's own generations.

Use with cot="full"

This LoRA is meant to run with YuE2's chain-of-thought on (cot="full"): the model first writes its own ABC score, then the music tokens conditioned on that score. It was trained with the score present half the time and absent half the time, so cot="off" still works, but in our renders the score-first path ended songs on its own 8 times out of 9 and sounded more deliberate, while the score-free path either bailed early or ran to the length cap. Set cot="full" in the pipeline call (ComfyUI: the mode input of the YuE2 nodes with a non-empty ABC step; our script: COT=full).

Files

file	what
ar_lora_inst_v3abc.safetensors	fp32, 392 named tensors: layers.{i}.self_attn.{q,k,v,o}_proj.lora_{A,B}, layers.{i}.mlp.{gate,up,down}_proj.lora_{A,B}. Apply W += lora_B @ lora_A, scale 1.0.
ar_lora_inst_v3abc.bf16.safetensors	same weights in bf16 (half size).
ar_lora_inst_v3abc_comfyui.safetensors	ComfyUI-native layout (fused qkv / gate_up keys), for the LoraLoader CLIP slot.
scripts/ar_generate.py, scripts/ckpt_io.py	reference merge-and-generate script (folds the LoRA into the base weights, then runs YuE2's stock pipeline).

Works with the stock NAR decoder. For real-production sounding output pair it with the NAR LoRA and tokenizer from yue2-mothersuperior-realaudio-tokenizer-v4 (our renders used that NAR LoRA).

ComfyUI

ar_lora_inst_v3abc_comfyui.safetensors is the same LoRA in ComfyUI's native YuE2 layout. In ComfyUI the AR planner lives in the CLIP slot, so load this file with the standard LoraLoader and connect its CLIP output onward (strength 1.0); the MODEL output is untouched by this file. Set the YuE2 Generate nodes' mode to full and connect the ABC node so the model writes its score first. ComfyUI fuses q/k/v into text_encoders.model.layers.{i}.self_attn.qkv_proj and gate/up into mlp.gate_up_proj; this file packs the separate LoRAs block-diagonally into those fused keys (lora_up/lora_down, no alpha = scale 1.0; up @ down equals the stacked deltas exactly). Pair with the NAR LoRA from yue2-mothersuperior-realaudio-tokenizer-v4 (nar_lora_joint_v4_comfyui.safetensors, MODEL slot) for real-production sound.

Prompting: three caption styles

The style field is a normal YuE2 tag string (genre, instruments, mood, BPM). The lyrics field carries the structure, in one of three forms the LoRA was trained on in equal thirds. Use only these bare tag names: intro, verse, pre-chorus, chorus, bridge, outro. One tag per line, real line breaks, nothing else in the brackets.

1. Bare — let the model choose the structure and length.

[instrumental]

2. Untimed tags — you choose the section order, the model chooses the timing.

[intro]
[verse]
[chorus]
[bridge]
[chorus]
[outro]

3. Timed tags — you also give each section a start and end in m:ss. The model was trained with exact section times from real tracks, so this is the strongest structural steer. Treat the times as a guide rather than a guarantee: the model follows the section order and proportions better than the absolute end time, and it tends toward 3–5 minute songs regardless of the plan.

[intro 0:00-0:15]
[verse 0:15-0:45]
[chorus 0:45-1:10]
[bridge 1:10-1:40]
[chorus 1:40-2:05]
[outro 2:05-2:30]

Keep production notes ([guitar solo, distorted]), lyrics text, and literal \n characters out of the lyrics field: the LoRA never saw them and they pull the output away from the trained behavior.

Example

COT=full STYLE_TEXT="dark ambient, water phone, theremin, piano, slow" \
python scripts/ar_generate.py ar_lora_inst_v3abc.safetensors none out_tag any_style_track plan.txt 21

plan.txt holds one of the three caption forms above. AR_SCALE (default 1.0) scales the LoRA delta; lower values give the base model more say.

Training notes

lr 6e-5, 5,000 steps, rank 64, one song per step with 2-step gradient accumulation; 50% of steps use the score-conditioned layout (cot=full) and 50% the score-free layout with a separate score-writing sequence, so the model always learns to write the score; an auxiliary section-cursor loss (frame → active tag line) and an up-weighted END token. Held-out loss on unseen instrumentals fell from 5.33 to 4.51 (score-free) and 3.64 to 2.92 (score-conditioned) with no overfitting turn. About 5% of tracks were held out and never trained on.

License

Weights derive from YuE2-3B (CC BY-NC 4.0): non-commercial use only. Provided as-is.