README.md

---
license: mit
license_link: https://huggingface.co/microsoft/Phi-4-multimodal-instruct/resolve/main/LICENSE
language:
  - en
tags:
  - mlx
  - apple-silicon
  - vision-language-model
  - multimodal
  - phi-4
  - quantized
  - 4bit
  - siglip
  - document-understanding
  - chart-understanding
  - ocr
pipeline_tag: image-text-to-text
library_name: mlx
base_model: microsoft/Phi-4-multimodal-instruct
datasets:
  - lmms-lab/DocVQA
  - lmms-lab/ai2d
  - MMMU/MMMU
  - HuggingFaceM4/ChartQA
  - lmms-lab/textvqa
  - echo840/OCRBench
  - derek-thomas/ScienceQA
  - AI4Math/MathVista
model-index:
  - name: Phi-4-multimodal-instruct-mlx-4bit
    results:
      - task:
          type: image-text-to-text
        dataset:
          name: AI2D
          type: lmms-lab/ai2d
          split: test
        metrics:
          - type: accuracy
            value: 83.0
            name: Accuracy (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: ChartQA
          type: HuggingFaceM4/ChartQA
          split: test
        metrics:
          - type: relaxed_accuracy
            value: 86.0
            name: Relaxed Accuracy (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: DocVQA
          type: lmms-lab/DocVQA
          split: validation
        metrics:
          - type: anls
            value: 82.8
            name: ANLS (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: TextVQA
          type: lmms-lab/textvqa
          split: validation
        metrics:
          - type: accuracy
            value: 80.0
            name: Accuracy (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: OCRBench
          type: echo840/OCRBench
          split: test
        metrics:
          - type: score
            value: 840
            name: Score/1000 (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: ScienceQA
          type: derek-thomas/ScienceQA
          split: test
        metrics:
          - type: accuracy
            value: 95.8
            name: Accuracy (n=48, image-only)
      - task:
          type: image-text-to-text
        dataset:
          name: MathVista
          type: AI4Math/MathVista
          split: testmini
        metrics:
          - type: accuracy
            value: 58.0
            name: Accuracy (n=100)
---

# Phi-4-Multimodal-Instruct — MLX 4-bit

A 4-bit quantized [Apple MLX](https://github.com/ml-explore/mlx) conversion of [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) for native inference on Apple Silicon.

**Converted by [Ferox AI](https://ferox.ca)** · Vision-language inference on MacBook / Mac Studio / Mac Pro without cloud dependencies.

| | |
|---|---|
| **Parameters** | 5.6B (pre-LoRA-fusion) |
| **Quantization** | 4-bit, group_size=64 (backbone only; SigLIP encoder remains FP16) |
| **Disk size** | ~3.9 GB |
| **Base model** | [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) |
| **License** | MIT |
| **Modality** | Vision + Text (Phase 1; audio deferred) |

> **Other variants:** [bf16 (full precision)](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-bf16) · [8-bit](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-8bit)

## Quickstart

```python
from mlx_vlm import load, generate

model, processor = load("ferox-ai/Phi-4-multimodal-instruct-mlx-4bit")

output = generate(
    model,
    processor,
    "Describe this image in detail.",
    ["path/to/image.jpg"],
    max_tokens=512,
    verbose=False,
)
print(output)
```

Requires `mlx-vlm >= 0.1.0` with Phi-4-MM architecture support. Install dependencies:

```bash
pip install mlx-vlm>=0.1.0 mlx>=0.22.0
```

## Benchmark Results

Evaluated with our internal evaluation harness on a single Apple Silicon device. Scores are computed on a 100-sample subset of each benchmark. Microsoft's reference scores are reported on the full dataset using PyTorch FP16 — direct comparison should account for both the precision difference and sample-size variance.

| Benchmark | This Model (4-bit) | bf16 | Microsoft FP16 (full dataset) | Metric |
|-----------|:------------------:|:----:|:-----------------------------:|--------|
| **AI2D** | 83.0 | 90.0 | 82.3 | Accuracy |
| **ChartQA** | **86.0** | 85.0 | 81.4 | Relaxed Accuracy |
| **DocVQA** | 82.8 | 86.2 | 93.2 | ANLS |
| **MathVista** | 58.0 | 58.0 | 62.4 | Accuracy |
| **MMMU** | 24.0 | 31.0 | 55.1 | Accuracy |
| **OCRBench** | 840 | 840 | 844 | Score / 1000 |
| **ScienceQA** | 95.8† | 100.0† | 97.5 | Accuracy |
| **TextVQA** | **80.0** | 82.0 | 75.6 | Accuracy |

† ScienceQA: 48 of 100 samples scored (image-bearing questions only; 52 text-only questions excluded).

### Quantization impact

Across all benchmarks, 4-bit quantization produces a mean accuracy delta of −2.2 percentage points relative to bf16 — within the expected range for 4-bit group quantization on a model of this scale.

### Note on MMMU

The 100-sample MMMU scores (24.0% 4-bit, 31.0% bf16) fall well below Microsoft's reported 55.1%. To isolate the cause, we ran a **full 900-sample MMMU validation** on the lossless bf16 variant and obtained **27.9%** — consistent with the subset, which confirms the gap is **not** caused by quantization or weight conversion. We were unable to reproduce Microsoft's 55.1% and attribute the difference to evaluation-harness and answer-extraction handling for MMMU's multiple-choice format (prompt formatting and option parsing), rather than to the model's underlying capability — which is better reflected by the document-, chart-, OCR-, and science-focused benchmarks above.

## Architecture

| Component | Details |
|-----------|---------|
| **Backbone** | Phi-4-Mini (3.8B) — 32 transformer layers, hidden_size=3072, 24 query heads / 8 KV heads (GQA), head_dim=128, LongRoPE positional encoding (131K context) |
| **Vision encoder** | SigLIP-SO400M NaViT — 27 layers, 16 heads, head_dim=72, hidden_size=1152 |
| **Vision projection** | 2-layer MLP: Linear(4608→3072) → GELU → Linear(3072→3072). Input is a 2×2 spatial merge of SigLIP patch features |
| **Vision LoRA** | rank=256, alpha=512 (~370M parameters) — **pre-fused** into backbone weights before quantization |
| **Image preprocessing** | Dynamic HD tiling (deterministic grid, up to 8 crops at 448×448). PIL + NumPy only; zero PyTorch dependency at inference |
| **Quantization** | 4-bit with group_size=64. Applied to backbone linear layers only; SigLIP encoder weights remain in FP16 |

### Weight provenance

Weights are converted from `microsoft/Phi-4-multimodal-instruct` using a deterministic pipeline:

1. Download source checkpoint (PyTorch safetensors)
2. Fuse vision LoRA adapters into backbone weights (eliminates runtime adapter overhead)
3. Remap weight keys to MLX naming conventions
4. Transpose LoRA matrices (PEFT → MLX format)
5. Quantize backbone to 4-bit (SigLIP excluded)
6. Serialize as MLX safetensors

The conversion and quantization pipeline is deterministic and fully reproducible from the base model.

## Intended Use

This model is designed for **local, on-device vision-language inference** on Apple Silicon hardware. Suitable applications include:

- Document understanding and extraction (invoices, forms, reports)
- Chart and diagram interpretation
- Visual question answering
- OCR and text recognition in images
- Educational content analysis

### Out of scope

- Audio processing (Phase 2, not included in this release)
- Production deployment without application-level safety filtering
- Use cases requiring guaranteed factual accuracy without human verification

## Limitations

- **100-sample evaluations.** Benchmark scores are computed on subsets, not full datasets. Expect variance relative to full-dataset evaluations.
- **Vision-only.** This is a Phase 1 release covering the vision modality. Audio support from the original Phi-4-multimodal architecture is not included.
- **No runtime LoRA switching.** Vision LoRA adapters are pre-fused; the model cannot dynamically swap adapters.
- **Apple Silicon required.** MLX is designed for Apple's unified memory architecture (M1/M2/M3/M4). This model will not run on CUDA or CPU-only systems.

## Citation

If you use this model in your work, please cite:

```bibtex
@misc{feroxai2026phi4mlx,
  title={Phi-4-Multimodal-Instruct MLX Conversion},
  author={Ferox AI},
  year={2026},
  url={https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-4bit},
  note={4-bit quantized MLX port of microsoft/Phi-4-multimodal-instruct}
}
```

## Acknowledgments

- **Microsoft Research** for the [Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) model and technical report
- **Apple MLX team** for the [MLX framework](https://github.com/ml-explore/mlx)
- **Prince Canuma** for [mlx-vlm](https://github.com/Blaizzy/mlx-vlm)