handover.md

Handover Log: NuExtract3 A100 HF Space Deployment

This document maintains a persistent, sequential log of all operations and states related to the deployment of the encryptd/ocr_vlm_nuextract3 Hugging Face Space.

Project Details

• Current Workspace Path: /Users/mdasad/hf_space_git/ocr_vlm_nuextract3
• Target Space ID: encryptd/ocr_vlm_nuextract3
• Hardware: A100 GPU (a100-large)
• Underlying Model: numind/NuExtract3 (based on Qwen3.5-4B)
• SDK: Custom Docker (Ubuntu 22.04 + CUDA 12.4.1 Devel)
• API URL: https://huggingface.co/proxy/encryptd-ocr-vlm-nuextract3.hf.space

---

Sequence of Events

1. Planning & Pre-checks (Completed)

• Verified active environment Hugging Face Token belongs to user encryptd.
• Researched model ID and verified numind/NuExtract3 is the active target.
• Created the new Space encryptd/ocr_vlm_nuextract3 on Hugging Face.

2. Transition to Custom Docker Space (Completed)

• Discovered that the default Gradio Space SDK enforces Python 3.13, which has no pre-compiled vllm wheels.
• Converted the Space to a custom Docker Space (sdk: docker) using nvidia/cuda:12.4.1-runtime-ubuntu22.04 and Python 3.10 to allow native GPU execution and fast wheel installation.

3. Triage of NVIDIA Driver Error (Completed)

• Problem: Upon container startup, the host GPU driver version was reported as 12040 (supporting up to CUDA 12.4). PyTorch (initially resolved to the latest 2.11.0 / CUDA 12.6/12.8 wheel) crashed with RuntimeError: The NVIDIA driver on your system is too old (found version 12040).
• Solution: Modified requirements.txt to explicitly use the official PyTorch CUDA 12.4 wheel repository (https://download.pytorch.org/whl/cu124) and pinned:
  • torch==2.5.1 (or torch==2.5.1+cu124 depending on system resolution)
  • vllm==0.7.2 This resolved the driver version conflict successfully due to CUDA's backward compatibility.

4. Triage of Tokenizer and Multimodal ValueError (Completed)

• Tokenizer Problem: The newer transformers>=5.x package has deprecated/removed the tokenizer attribute all_special_tokens_extended. Sourcing an older version of vLLM (like v0.8.0 or below) with transformers>=5.x caused the AttributeError: TokenizersBackend has no attribute all_special_tokens_extended crash.
• Multimodal ValueError / KeyError 'qwen3_5': Older vLLM/Transformers versions either throw ValueError: limit_mm_per_prompt is only supported for multimodal models (due to missing remote-code registration) or KeyError: 'qwen3_5' (due to the qwen3_5 architecture being unrecognized in older transformers libraries).
• Unified Resolution:
  • Upgraded to vllm==0.12.0 and torch==2.6.0+cu124.
  • Sourced the CUDA 12.4 index (cu124) to guarantee driver compatibility with 12040 on HF Spaces.
  • Sourced vllm==0.12.0 which natively targets transformers>=5.x and supports the qwen3_5 architecture without tokenizer or registration issues.
  • Restored all high-performance model card recommended arguments:
    • --limit-mm-per-prompt '{"image": 99, "video": 0}' (allows multi-page document processing up to 99 pages/images).
    • --chat-template-content-format openai and --generation-config vllm (properly registers the Qwen 3.5 architecture).
    • --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}' (enables multi-token speculative decoding).
    • --enforce-eager.

5. First Push & Build (Completed)

• Configured Hugging Face login authentication on the local CLI, storing Git credentials in the osxkeychain helper, and cleaned the git remote URL to remove the embedded token, strictly referencing it via environment credentials.
• Pushed the finalized configuration.

6. Resolution of the NVIDIA Driver 12040 Mismatch (Completed)

• Problem: When pip installed the standard torch package, it fell back to PyPI's default CUDA 13.0/12.8 wheel because the https://download.pytorch.org/whl/cu124 index contains 2.6.0+cu124 (requiring the exact version suffix match to prevent PyPI fallback). This mismatch crashed the container runtime on the Space's A100 GPU host (which runs NVIDIA driver 12040 supporting up to CUDA 12.4).
• Resolution:
  • Updated requirements.txt to explicitly request torch==2.6.0+cu124.
  • Added a direct pre-installation layer in the Dockerfile targeting --index-url https://download.pytorch.org/whl/cu124 to absolutely guarantee that PyTorch installs the CUDA 12.4 pre-compiled wheel.
  • This guarantees PyTorch matches the host's driver version (12040) perfectly while allowing the modern vllm==0.12.0 runtime to execute natively on the A100 hardware, dynamically pulling its compatible torchvision version automatically.

---

Current Status

• Hugging Face Status: stage: RUNNING
• The FastAPI API proxy is live and bound to the Gradio web interface at the root.
• The vLLM engine is fully operational with complete multi-page document image processing and multi-token speculative decoding enabled.