Nihonga-v1.0

Qwen-Image-2.1 in 10 steps instead of 40: about 3.9 times faster (1.75 s against 6.89 s for a 1024 × 1024 image on one H100), with two of its 32 transformer blocks replaced by small learned bridges, and 15 of 16 exact spellings on our hardest sign test (the original: 16). Stock diffusers plus the loader file shipped in this repo.

Nihonga (日本画) is the Japanese painting tradition of mineral pigments on silk and paper, built up in few, deliberate layers. This model applies that spirit to an image transformer: fewer blocks, and far fewer strokes to finish an image.

A paper with the full method and measurements is coming soon. Built with Qwen.

Project page: niwakiai.com/nihonga · full research log, with every experiment and failure: josejuanmartinez.github.io/research/nihonga · notebooks with all code and results: github.com/josejuanmartinez/nihonga

What it is

Qwen-Image-2.1's transformer with blocks 5 and 20 removed. Each is replaced by a bridge of about 25 M weights (against 218 M for a block): a timestep-conditioned MLP plus an 8-head attention branch with its own prompt cache, trained end to end against the original model's output. On top, a Lightning LoRA that we trained for this model (rank 32, on the attention of the remaining 30 blocks) is fused into the weights, so the model renders in 10 steps without guidance.

Spelling breaks first when a diffusion transformer is pruned or distilled, and average losses hardly see it: letters cover a few percent of an image. Blocks were therefore chosen by checking text, not only average error; the Lightning LoRA was trained with the tokens that hold letters (found by OCR) weighted ten times; and the sampler's first jump from noise, where the layout is decided, is split into three small steps.

The text encoder (17.5 GB) and the VAE are unchanged and are not in this repo: they load from Qwen/Qwen-Image-2.1.

Benchmarks

Same prompts and seeds for every row: 5 text prompts with 4 seeds each (a storefront sign, a shop-name sign, a poster tagline, a handwritten Chinese note, a banner) and 25 general prompts, judged at full resolution. Speed: batch 1, 1024 × 1024, one H100, BF16, medians; the pruned model at 40 steps was timed in a separate run (the original measured 6.80 s there).

model	blocks	steps	time per image ↓	spelling, 16 items ↑	general images unlike its 40-step source ↓
Qwen-Image-2.1	32	40	6.89 s	16	—
pruned model, no LoRA	30 + 2 bridges	40	6.56 s	15	—
pruned model, earlier 8-step LoRA (trained on the original's images)	30 + 2 bridges	8	—	12	5 of 25
Lightning LoRA on the full model (LightX2V, published)	32	4	0.87 s	13	—
this model	30 + 2 bridges	10	1.75 s	15	2 of 25

The two general images that differ: a laptop lighting a face is missing, and a group of four friends has five people. Compositions are not seed-for-seed copies of the original model's: the gentle start and the Lightning LoRA lead to different, clean pictures for the same noise.

Left to right in each group: Qwen-Image-2.1 (40 steps), the published LightX2V Lightning LoRA (4 steps; note "Acchor") and Nihonga v1.0 (10 steps); same prompt and seed in each row.

Architecture

The Lightning LoRA

A LoRA leaves each weight matrix frozen and learns a thin correction beside it: two small matrices B (4,096 × 32) and A (32 × 4,096), so a layer computes W·x + B·(A·x). Ours sits on q, k, v and out of the 30 blocks: 120 pairs, 31.5 M trainable weights.

It was trained by step distillation. The pruned model was run in its normal 40 steps on 1,000 prompts (500 with text), saving the image after every step, and the LoRA learned to make one big step land where five of those small steps land. Letters are a few percent of the tokens, so an OCR detector marked the tokens that hold text in each training image and those count ten times in the loss; 3,000 updates on one H100. At use, the first jump out of noise is split into three small steps, then seven big LoRA steps: 10 in all.

After training each pair was multiplied out and added to its matrix, W′ = W + B·A (computed in FP32, stored in BF16). The LoRA is fused: this repo has no separate LoRA file, and it costs no extra weights or time.

Model dimensions

transformer parameters	6.729 B (from 7.115 B)
transformer blocks	30 of 32, plus 2 bridges in slots 5 and 20 (8 heads × 128, MLP rank 1024; ~25 M each)
fused Lightning LoRA	rank 32, alpha 32, on q, k, v and out of the 30 blocks (31.5 M weights before fusing)
precision	BF16, 13.46 GB in 3 shards
sampling	10 steps, no classifier-free guidance (true_cfg_scale=1.0)
schedule	points 0, 1, 3, 5, 10, 15, 20, 25, 30, 35 of the stock 40-step schedule, then the end
memory	39.3 GB peak at 1024 × 1024 with the text encoder and VAE on the GPU (original: 39.6 GB)
modality	text to image (editing, reference images and transparency not trained or tested)

Usage (diffusers)

Requires the diffusers commit with the Qwen-Image-2.1 pipeline, and nihonga.py from this repo: it defines the bridges, loads the transformer, and provides the 10-step schedule.

pip install "diffusers @ git+https://github.com/huggingface/diffusers@bdc2bea37a36038c44452811610489ea30ede229" \
    transformers accelerate safetensors huggingface_hub torch
hf download jjmcarrascosa/Nihonga-v1.0 --local-dir ./nihonga-v1

import sys, torch
sys.path.insert(0, "./nihonga-v1")
from nihonga import load_pipeline           # nihonga.py from this repo

pipe = load_pipeline("./nihonga-v1")        # Nihonga transformer + Qwen-Image-2.1 text encoder, VAE and processor
image = pipe('A storefront named "The Rusty Anchor", warm window light', width=1024, height=1024,
             num_inference_steps=10, true_cfg_scale=1.0,
             generator=torch.Generator("cpu").manual_seed(0)).images[0]
image.save("nihonga.png")

The 10-step schedule cannot be produced by the stock scheduler on its own. There are two ways to run it; they give pixel-identical images.

• Scheduler class (default). load_pipeline installs NihongaScheduler, which builds the stock 40-step schedule for the image size and keeps the ten points. Call the pipeline with num_inference_steps=10.

• Plain sigmas, for the stock scheduler and for interfaces with a custom-sigmas input. Make the stock scheduler neutral (no shift, no terminal stretch), because the values already include them:

  from nihonga import use_plain_sigmas, nihonga_sigmas
  config = use_plain_sigmas(pipe)
  image = pipe(prompt, width=w, height=h, sigmas=nihonga_sigmas(config, w, h), true_cfg_scale=1.0).images[0]
  

  sigmas.json lists the ten values for 1024×1024, 1376×768, 768×1376, 1184×896, 896×1184, 1248×832 and 832×1248 (the scheduler appends the final 0), for ComfyUI and other interfaces without code.

example.py runs both.

Notes and limitations

• Compression trades quality: on our 25 general prompts, 2 images miss content the 40-step pruned model draws (a small object, an exact count). Small objects and exact counts are the weakest point at 10 steps.
• Text: 15 of 16 exact on our hardest signs; a long word in a stylized font is the usual miss.
• Speed comes almost entirely from the 10-step schedule; removing two blocks saves about 4% per step.
• Memory hardly changes: the unchanged text encoder holds 17.5 GB. Quantization is not part of v1.0.
• Evaluated on a small set of prompts (45 held-out images). The base model's biases are inherited.
• Text to image only: Qwen-Image-2.1 also edits images, uses up to 10 reference images and makes transparent images; none of that was trained or tested here.
• Non-commercial use only (Qwen Research License).

License

Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT, Copyright (c) 2026 Hangzhou Tongyi Laboratory Technology Co., Ltd. All Rights Reserved. Nihonga-v1.0 is a modified version of the Qwen-Image-2.1 transformer, for non-commercial use; the modified files are listed in NOTICE.

Base model by the Qwen team (see license); pruning, bridges, distillation and the 10-step schedule by the Niwaki project, 2026-10. Method details: paper coming soon.