Hayai OCR v2.1

Hayai is a lightweight (~150M parameter) vision-to-text OCR model designed for fast, crop-level transcription across Japanese, Chinese, Korean, and English.

By pairing Google’s SigLIP2 NaFlex vision encoder with a 12-layer custom causal transformer decoder, Hayai reads dense, stylized, horizontal, and vertical text directly from images in a single forward pass without requiring a separate text detection stage (e.g., DBNet/YOLO). (Doesn't work for full pages though. Only crops)

What's New in v2.1 (Joint Multimodal + Linguistic Pretraining)

In prior versions, compact OCR models struggled with visually ambiguous CJK radicals and homoglyphs (e.g., confusing 銀 vs. 高 or 校 vs. 枚) because a pure image-trained decoder lacked statistical language priors.

Hayai v2.1 introduces Joint Multi-Task Training:

Zero-Overhead Language Prior Injection: Co-trained directly on streaming Wikipedia (JA, ZH, KO, EN) and Aozora Bunko corpora. Text-only passes enter the decoder directly with 1D RoPE (bypassing the vision backbone), teaching the decoder deep contextual CJK transition probabilities.
Radical & Counter Disambiguation: Eliminates homograph and counter errors on complex layouts.
Flawless Multi-Script Support: Significantly boosted English accuracy while maintaining high precision on vertical Japanese, Korean Hangul, and Chinese Hanzi.

Architecture

Total Parameters: ~150M
Vision Encoder: google/siglip2-base-patch16-naflex (~86M params)
- Native aspect ratio preservation via NaFlex patching (no forced warping/squashing of dense characters).
Projector: 2-layer MLP mapping visual patch features into the decoder hidden dimension.
Decoder: 12-layer Causal Transformer (~60M params)
- Attention: Grouped-Query Attention (8 query heads, 2 key/value heads) with RMSNorm on queries & keys.
- FFN: SwiGLU feed-forward layers (d_model = 512, d_ffn = 2048).
- Positional Embeddings: Dynamic 2D Multimodal Rotary Position Embeddings (2D mRoPE) over visual tokens; 1D RoPE over text tokens.
- Attention Masking: Block-causal attention (bidirectional among visual patch tokens, causal across output text tokens).

Usage

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor, PreTrainedTokenizerFast

# Load Model, Processor & Tokenizer
MODEL_ID = "JustANormalTinkerer/hayai-ocr-v2"
model = AutoModel.from_pretrained(MODEL_ID, trust_remote_code=True).cuda().eval()
tokenizer = PreTrainedTokenizerFast.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-naflex")

# Load and Preprocess Image
image = Image.open("example.png").convert("RGB")
# Use max_num_patches=256 for standard lines; 384 or 512 for dense/complex panels
inputs = processor(images=[image], max_num_patches=256, return_tensors="pt").to("cuda")

with torch.no_grad():
    texts = model.generate(
        pixel_values=inputs["pixel_values"],
        pixel_attention_mask=inputs["pixel_attention_mask"],
        spatial_shapes=inputs["spatial_shapes"],
        tokenizer=tokenizer,
        max_new_tokens=128,
        num_beams=4,              # Recommended: 3-4 beams for optimal CJK disambiguation
        repetition_penalty=1.0,   # Keep at 1.0 (disabled) for OCR accuracy
    )

print(texts[0])

Note: trust_remote_code=True is required because the model utilizes custom block-causal attention and 2D mRoPE definitions (configuration_hayai.py, modeling_hayai.py).

Benchmarks

JMangaBench_Mixed

Model	CER ↓	Exact Match ↑	Text-only CER ↓	Text-only Exact Match ↑
MangaOCR	4.683%	73.524%	2.700%	82.867%
HayaiOCR	6.738%	71.272%	4.967%	80.949%
HayaiOCR-v2	4.534%	73.645%	2.872%	82.227%
HayaiOCR-v2.1	3.225%	79.671%	1.896%	87.461%
BaberuOCR	4.589%	72.246%	2.603%	81.649%
PaddleOCR-VL-0.9B-For-Manga	2.910%	78.911%	1.866%	84.662%

Fine-tuning Dataset Train Split (ZH + JA/KO Onomatopoeia + EN)

Model Name	Mean CER ↓	Throughput on L4 GPU (FPS) ↑
Hayai OCR v2	8.52%	37.25
PaddleOCR-VL-For-Manga	24.66%	3.60

Private Pretraining Dataset Train Split: CJK

Model Name	Mean CER ↓	Throughput on L4 GPU (FPS) ↑
Hayai OCR v2	10.56%	31.95
Hayai OCR v2.1	12.94%	54.22*
PaddleOCR-VL-For-Manga	38.69%	2.22

*Throughput gain in v2.1 is due to optimized decoding batching and kernel fusion.

Hayai OCR matches or outperforms 0.9B parameter models while delivering 10× higher throughput and operating within a ~300MB VRAM footprint in FP16.

Training Methodology

Training was conducted in two coordinated phases using Kaggle 2× NVIDIA T4 GPUs.

1. Multi-Task Joint Base Training (~19,000 Steps)

OCR Stream: JustANormalTinkerer/hayai-dataset-merged (~1M images) streamed and sharded across GPUs.
Linguistic Prior Stream: Interleaved token-packed streams from Japanese, Chinese, Korean, and English Wikipedia alongside the Aozora Bunko clean literature dataset.
Loss Objective:
```
L_total = L_ocr + 0.30 * L_text
```

2. Optimization

Optimizer: Muon for decoder 2D weight matrices (orthogonalized momentum updates); AdamW for 1D vectors, embeddings, norms, and the SigLIP2 vision encoder.
Learning Rates: Base LR 8e-5 (Muon / Decoder AdamW) and 1e-5 (Vision AdamW), decayed via cosine schedule with a 5% linear warmup.
Augmentation: Random affine transforms, perspective shifts, subtle rotation (±6°), color jitter, blur, and sharpness adjustment.
Precision: Mixed Precision (FP16) with dynamic gradient scaling.

Text Normalization

For consistent Character Error Rate (CER) reproduction and downstream evaluation, text should be normalized as follows:

import re
import unicodedata

def normalize_text(text: str) -> str:
    if not text:
        return ""
    text = unicodedata.normalize("NFKC", str(text))
    text = re.sub(r'[\r\n\t]+', ' ', text)
    # Remove space only between CJK characters
    cjk_char = r'[\u4e00-\u9fff\u3040-\u30ff\u3400-\u4dbf\uac00-\ud7af]'
    text = re.sub(f'({cjk_char})\\s+({cjk_char})', r'\1\2', text)
    return re.sub(r'\s+', ' ', text).strip()

Best Practices & Limitations

Repetition Penalty: Keep repetition_penalty = 1.0. Penalties > 1.0 force the model to avoid valid repeated characters (e.g., 2校 ... 1校 or 学校).

Downloads last month: 1,151

Safetensors

Model size

0.2B params

Tensor type

F32

JustANormalTinkerer
/

hayai-ocr-v2