README.md

metadata

license: mit
license_link: >-
  https://huggingface.co/microsoft/Phi-4-multimodal-instruct/resolve/main/LICENSE
language:
  - en
tags:
  - mlx
  - apple-silicon
  - vision-language-model
  - multimodal
  - phi-4
  - quantized
  - 4bit
  - siglip
  - document-understanding
  - chart-understanding
  - ocr
pipeline_tag: image-text-to-text
library_name: mlx
base_model: microsoft/Phi-4-multimodal-instruct
datasets:
  - lmms-lab/DocVQA
  - lmms-lab/ai2d
  - MMMU/MMMU
  - HuggingFaceM4/ChartQA
  - lmms-lab/textvqa
  - echo840/OCRBench
  - derek-thomas/ScienceQA
  - AI4Math/MathVista
model-index:
  - name: Phi-4-multimodal-instruct-mlx-4bit
    results:
      - task:
          type: image-text-to-text
        dataset:
          name: AI2D
          type: lmms-lab/ai2d
          split: test
        metrics:
          - type: accuracy
            value: 83
            name: Accuracy (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: ChartQA
          type: HuggingFaceM4/ChartQA
          split: test
        metrics:
          - type: relaxed_accuracy
            value: 86
            name: Relaxed Accuracy (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: DocVQA
          type: lmms-lab/DocVQA
          split: validation
        metrics:
          - type: anls
            value: 82.8
            name: ANLS (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: TextVQA
          type: lmms-lab/textvqa
          split: validation
        metrics:
          - type: accuracy
            value: 80
            name: Accuracy (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: OCRBench
          type: echo840/OCRBench
          split: test
        metrics:
          - type: score
            value: 840
            name: Score/1000 (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: ScienceQA
          type: derek-thomas/ScienceQA
          split: test
        metrics:
          - type: accuracy
            value: 95.8
            name: Accuracy (n=48, image-only)
      - task:
          type: image-text-to-text
        dataset:
          name: MathVista
          type: AI4Math/MathVista
          split: testmini
        metrics:
          - type: accuracy
            value: 58
            name: Accuracy (n=100)

Phi-4-Multimodal-Instruct — MLX 4-bit

A 4-bit quantized Apple MLX conversion of microsoft/Phi-4-multimodal-instruct for native inference on Apple Silicon.

Converted by Ferox AI · Vision-language inference on MacBook / Mac Studio / Mac Pro without cloud dependencies.

Parameters	5.6B (pre-LoRA-fusion)
Quantization	4-bit, group_size=64 (backbone only; SigLIP encoder remains FP16)
Disk size	~3.9 GB
Base model	microsoft/Phi-4-multimodal-instruct
License	MIT
Modality	Vision + Text (Phase 1; audio deferred)

Other variants: bf16 (full precision) · 8-bit

Quickstart

from mlx_vlm import load, generate

model, processor = load("ferox-ai/Phi-4-multimodal-instruct-mlx-4bit")

output = generate(
    model,
    processor,
    "Describe this image in detail.",
    ["path/to/image.jpg"],
    max_tokens=512,
    verbose=False,
)
print(output)

Requires mlx-vlm >= 0.1.0 with Phi-4-MM architecture support. Install dependencies:

pip install mlx-vlm>=0.1.0 mlx>=0.22.0

Benchmark Results

Evaluated with our internal evaluation harness on a single Apple Silicon device. Scores are computed on a 100-sample subset of each benchmark. Microsoft's reference scores are reported on the full dataset using PyTorch FP16 — direct comparison should account for both the precision difference and sample-size variance.

Benchmark	This Model (4-bit)	bf16	Microsoft FP16 (full dataset)	Metric
AI2D	83.0	90.0	82.3	Accuracy
ChartQA	86.0	85.0	81.4	Relaxed Accuracy
DocVQA	82.8	86.2	93.2	ANLS
MathVista	58.0	58.0	62.4	Accuracy
MMMU	24.0	31.0	55.1	Accuracy
OCRBench	840	840	844	Score / 1000
ScienceQA	95.8†	100.0†	97.5	Accuracy
TextVQA	80.0	82.0	75.6	Accuracy

† ScienceQA: 48 of 100 samples scored (image-bearing questions only; 52 text-only questions excluded).

Quantization impact

Across all benchmarks, 4-bit quantization produces a mean accuracy delta of −2.2 percentage points relative to bf16 — within the expected range for 4-bit group quantization on a model of this scale.

Note on MMMU

The 100-sample MMMU scores (24.0% 4-bit, 31.0% bf16) fall well below Microsoft's reported 55.1%. To isolate the cause, we ran a full 900-sample MMMU validation on the lossless bf16 variant and obtained 27.9% — consistent with the subset, which confirms the gap is not caused by quantization or weight conversion. We were unable to reproduce Microsoft's 55.1% and attribute the difference to evaluation-harness and answer-extraction handling for MMMU's multiple-choice format (prompt formatting and option parsing), rather than to the model's underlying capability — which is better reflected by the document-, chart-, OCR-, and science-focused benchmarks above.

Architecture

Component	Details
Backbone	Phi-4-Mini (3.8B) — 32 transformer layers, hidden_size=3072, 24 query heads / 8 KV heads (GQA), head_dim=128, LongRoPE positional encoding (131K context)
Vision encoder	SigLIP-SO400M NaViT — 27 layers, 16 heads, head_dim=72, hidden_size=1152
Vision projection	2-layer MLP: Linear(4608→3072) → GELU → Linear(3072→3072). Input is a 2×2 spatial merge of SigLIP patch features
Vision LoRA	rank=256, alpha=512 (~370M parameters) — pre-fused into backbone weights before quantization
Image preprocessing	Dynamic HD tiling (deterministic grid, up to 8 crops at 448×448). PIL + NumPy only; zero PyTorch dependency at inference
Quantization	4-bit with group_size=64. Applied to backbone linear layers only; SigLIP encoder weights remain in FP16

Weight provenance

Weights are converted from microsoft/Phi-4-multimodal-instruct using a deterministic pipeline:

1. Download source checkpoint (PyTorch safetensors)
2. Fuse vision LoRA adapters into backbone weights (eliminates runtime adapter overhead)
3. Remap weight keys to MLX naming conventions
4. Transpose LoRA matrices (PEFT → MLX format)
5. Quantize backbone to 4-bit (SigLIP excluded)
6. Serialize as MLX safetensors

The conversion and quantization pipeline is deterministic and fully reproducible from the base model.

Intended Use

This model is designed for local, on-device vision-language inference on Apple Silicon hardware. Suitable applications include:

• Document understanding and extraction (invoices, forms, reports)
• Chart and diagram interpretation
• Visual question answering
• OCR and text recognition in images
• Educational content analysis

Out of scope

• Audio processing (Phase 2, not included in this release)
• Production deployment without application-level safety filtering
• Use cases requiring guaranteed factual accuracy without human verification

Limitations

• 100-sample evaluations. Benchmark scores are computed on subsets, not full datasets. Expect variance relative to full-dataset evaluations.
• Vision-only. This is a Phase 1 release covering the vision modality. Audio support from the original Phi-4-multimodal architecture is not included.
• No runtime LoRA switching. Vision LoRA adapters are pre-fused; the model cannot dynamically swap adapters.
• Apple Silicon required. MLX is designed for Apple's unified memory architecture (M1/M2/M3/M4). This model will not run on CUDA or CPU-only systems.

Citation

If you use this model in your work, please cite:

@misc{feroxai2026phi4mlx,
  title={Phi-4-Multimodal-Instruct MLX Conversion},
  author={Ferox AI},
  year={2026},
  url={https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-4bit},
  note={4-bit quantized MLX port of microsoft/Phi-4-multimodal-instruct}
}

Acknowledgments

• Microsoft Research for the Phi-4-multimodal-instruct model and technical report
• Apple MLX team for the MLX framework
• Prince Canuma for mlx-vlm