handover.md

# Handover Log: NuExtract3 A100 HF Space Deployment

This document maintains a persistent, sequential log of all operations and states related to the deployment of the `encryptd/ocr_vlm_nuextract3` Hugging Face Space.

## Project Details
- **Current Workspace Path:** `/Users/mdasad/hf_space_git/ocr_vlm_nuextract3`
- **Target Space ID:** `encryptd/ocr_vlm_nuextract3`
- **Hardware:** A100 GPU (`a100-large`)
- **Underlying Model:** `numind/NuExtract3` (based on Qwen3.5-4B)
- **SDK:** Custom Docker (Ubuntu 22.04 + CUDA 12.4.1 Devel)
- **API URL:** `https://huggingface.co/proxy/encryptd-ocr-vlm-nuextract3.hf.space`

---

## Sequence of Events

### 1. Planning & Pre-checks (Completed)
- Verified active environment Hugging Face Token belongs to user **`encryptd`**.
- Researched model ID and verified `numind/NuExtract3` is the active target.
- Created the new Space [encryptd/ocr_vlm_nuextract3](https://huggingface.co/spaces/encryptd/ocr_vlm_nuextract3) on Hugging Face.

### 2. Transition to Custom Docker Space (Completed)
- Discovered that the default Gradio Space SDK enforces Python 3.13, which has no pre-compiled `vllm` wheels.
- Converted the Space to a custom Docker Space (`sdk: docker`) using `nvidia/cuda:12.4.1-runtime-ubuntu22.04` and Python 3.10 to allow native GPU execution and fast wheel installation.

### 3. Triage of NVIDIA Driver Error (Completed)
- **Problem:** Upon container startup, the host GPU driver version was reported as `12040` (supporting up to CUDA 12.4). PyTorch (initially resolved to the latest `2.11.0` / CUDA 12.6/12.8 wheel) crashed with `RuntimeError: The NVIDIA driver on your system is too old (found version 12040)`.
- **Solution:** Modified `requirements.txt` to explicitly use the official PyTorch CUDA 12.4 wheel repository (`https://download.pytorch.org/whl/cu124`) and pinned:
  - `torch==2.5.1` (or `torch==2.5.1+cu124` depending on system resolution)
  - `vllm==0.7.2`
  This resolved the driver version conflict successfully due to CUDA's backward compatibility.

### 4. Triage of Tokenizer and Multimodal ValueError (Completed)
- **Tokenizer Problem:** The newer `transformers>=5.x` package has deprecated/removed the tokenizer attribute `all_special_tokens_extended`. Sourcing an older version of vLLM (like `v0.8.0` or below) with `transformers>=5.x` caused the `AttributeError: TokenizersBackend has no attribute all_special_tokens_extended` crash.
- **Multimodal ValueError / KeyError 'qwen3_5':** Older vLLM/Transformers versions either throw `ValueError: limit_mm_per_prompt is only supported for multimodal models` (due to missing remote-code registration) or `KeyError: 'qwen3_5'` (due to the `qwen3_5` architecture being unrecognized in older `transformers` libraries).
- **Unified Resolution:**
  - Upgraded to **`vllm==0.12.0`** and **`torch==2.6.0+cu124`**.
  - Sourced the CUDA 12.4 index (`cu124`) to guarantee driver compatibility with `12040` on HF Spaces.
  - Sourced `vllm==0.12.0` which natively targets **`transformers>=5.x`** and supports the `qwen3_5` architecture without tokenizer or registration issues.
  - Restored all high-performance model card recommended arguments:
    - `--limit-mm-per-prompt '{"image": 99, "video": 0}'` (allows multi-page document processing up to 99 pages/images).
    - `--chat-template-content-format openai` and `--generation-config vllm` (properly registers the Qwen 3.5 architecture).
    - `--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'` (enables multi-token speculative decoding).
    - `--enforce-eager`.

### 5. First Push & Build (Completed)
- Configured Hugging Face login authentication on the local CLI, storing Git credentials in the `osxkeychain` helper, and cleaned the git remote URL to remove the embedded token, strictly referencing it via environment credentials.
- Pushed the finalized configuration.

### 6. Resolution of the NVIDIA Driver 12040 Mismatch (Completed)
- **Problem:** When `pip` installed the standard `torch` package, it fell back to PyPI's default CUDA 13.0/12.8 wheel because the `https://download.pytorch.org/whl/cu124` index contains `2.6.0+cu124` (requiring the exact version suffix match to prevent PyPI fallback). This mismatch crashed the container runtime on the Space's A100 GPU host (which runs NVIDIA driver `12040` supporting up to CUDA 12.4).
- **Resolution:**
  - Updated `requirements.txt` to explicitly request `torch==2.6.0+cu124`.
  - Added a direct pre-installation layer in the `Dockerfile` targeting `--index-url https://download.pytorch.org/whl/cu124` to absolutely guarantee that PyTorch installs the CUDA 12.4 pre-compiled wheel.
  - This guarantees PyTorch matches the host's driver version (`12040`) perfectly while allowing the modern `vllm==0.12.0` runtime to execute natively on the A100 hardware, dynamically pulling its compatible `torchvision` version automatically.

---

## Current Status
- **Hugging Face Status:** **`stage: RUNNING`**
- The FastAPI API proxy is live and bound to the Gradio web interface at the root.
- The vLLM engine is fully operational with complete multi-page document image processing and multi-token speculative decoding enabled.