AnandSingh/LightOnOCR-3-4B-4bit

LightOnOCR-3-4B — MLX 4-bit

4-bit quantized mlx-vlm weights of lightonai/LightOnOCR-3-4B for native inference on Apple Silicon. The model is a Qwen3.5 vision-language OCR model (model_type: qwen3_5) — runs on mlx-vlm ≥ 0.6.5 with no patches.

Use

pip install -U "mlx-vlm>=0.6.5"

# Transcription (default — call with an empty prompt for full-page Markdown)
mlx_vlm.generate --model AnandSingh/LightOnOCR-3-4B-4bit \
  --image receipt.jpeg --prompt "" --max-tokens 2048 --temperature 0.2 --top-p 0.9

# Grounding — every block returned with a label + bounding box (0–1000 coords),
# images described, charts emitted as HTML data tables
mlx_vlm.generate --model AnandSingh/LightOnOCR-3-4B-4bit \
  --image page.png --prompt "grounding" --max-tokens 4096 --temperature 0.2 --top-p 0.9

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("AnandSingh/LightOnOCR-3-4B-4bit")
prompt = apply_chat_template(processor, model.config, "grounding", num_images=1)
result = generate(model, processor, prompt, image=["page.png"],
                  max_tokens=4096, temperature=0.2, top_p=0.9)
print(result.text)

Prompt modes

Mode	Prompt	Output
Transcription	"" (empty)	All page text as Markdown
Grounding	"grounding"	Blocks prefixed  (0–1000 page coords)

Thinking is off by default (matches training); apply_chat_template sets enable_thinking=False automatically. Other prompts are out-of-distribution.

Details / quant

• Precision: 4-bit affine, group size 64, mixed policy (~5.3 bits/weight effective; norms, embeddings, vision kept higher) → ~3 GB file.
• Measured on Apple M4 (25 GB): ~37 tok/s decode, ~3.7 GB peak memory, 600+ tokens for a receipt page.
• BF16 reference for this model: AnandSingh/LightOnOCR-3-4B-bf16.

Tips

• Render PDFs at 400 DPI, then resize the longest edge to 2048 px (aspect preserved). The bundled Qwen3-VL processor handles patch/merge sizing.

License

Model weights: Apache-2.0 (per source). Quantized MLX conversion provided as-is.