README.md

metadata

pipeline_tag: visual-document-retrieval
library_name: transformers
language:
  - multilingual
license: other
license_name: webai-non-commercial-license-v1.0
license_link: https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b/blob/main/LICENSE.md
base_model: Qwen/Qwen3.5-4B
datasets:
  - vidore/colpali_train_set
  - Tevatron/docmatix-ir
  - openbmb/VisRAG-Ret-Train-In-domain-data
  - openbmb/VisRAG-Ret-Train-Synthetic-data
  - llamaindex/vdr-multilingual-train
  - Tevatron/wiki-ss-nq
tags:
  - text
  - image
  - multimodal-embedding
  - visual-document-retrieval
  - vidore
  - colbert
  - colqwen3_5
  - multilingual-embedding

webAI-Official/webAI-ColVec1.1-4b

⚡ Summary

webAI-Official/webAI-ColVec1.1-4b is a ColBERT-style multimodal embedding model based on Qwen/Qwen3.5-4B. It maps text queries and visual documents (images or rendered PDF pages) into aligned, L2-normalized multi-vector embeddings for late-interaction retrieval.

The model uses bidirectional attention in Qwen3.5's full-attention layers and a learned 640-dimensional projection head. The unused language-model head has been removed from the released checkpoint.

Training data

We created filtered, balanced, and multilingual curated subsets from the six public datasets below. This 4B model was trained on a 500,000-sample curated subset. The companion 9B-backbone model was trained on a separate 750,000-sample curated subset.

Dataset source
ColPali Train Set
Docmatix-IR
VisRAG In-Domain
VisRAG Synthetic
VDR Multilingual Train
Wiki-SS-NQ

🛠️ Model specifications

Feature	Detail
Architecture	Qwen3.5-4B vision-language model + 640-dimensional linear projection
Released parameters	4,540,904,576
Method	ColBERT-style late interaction with MaxSim scoring
Output	L2-normalized multi-vector embeddings (sequence_length, 640)
Modalities	Text queries and document images
Attention	Bidirectional full-attention layers; selectable FlashAttention 2, FlashAttention 3, or SDPA kernel
Visual-token budget	1,792 tokens per image in the released processor
Training	LoRA adapters and a fully trained projection layer, merged for release
Weights	bfloat16; language-model head removed

Key properties

• Unified encoder: The same model encodes text and document images.
• Token-level retrieval: Multi-vector embeddings preserve fine-grained layout and content signals that single-vector pooling can discard.
• Compact projection: Hidden states are projected to 640 dimensions without an activation function.
• Bidirectional retrieval attention: Selecting FlashAttention 2 or 3 changes the execution kernel, not the model's bidirectional attention mode.

📊 Evaluation results

The table reports NDCG@10 × 100 on the eight public ViDoRe V3 tasks. Private Nuclear and Telecom scores are intentionally omitted and will be added after official evaluation. Each task value is the mean of its six language subsets; the public average is the unweighted mean of the eight public task values.

The result artifacts record MTEB 2.18.5, Transformers 5.14.1, PyTorch 2.9.0 with CUDA 12.8, bfloat16, FlashAttention 2.8.3, and batch size 32. The release processor is configured with a 1,792 visual-token budget. Comparator values were read from the live ViDoRe V3 MTEB leaderboard on July 22, 2026.

Model	CompSci	Energy	FinanceEn	FinanceFr	HR	Industrial	Pharma	Physics	Avg. public
webAI-ColVec1.1-8b	80.14	70.09	71.89	55.10	68.49	57.41	67.88	51.29	65.29
VultronRetriever Prime	79.81	70.26	69.01	54.51	66.82	57.41	68.19	51.73	64.72
webAI-ColVec1-9b	80.92	69.77	68.28	53.72	70.04	57.18	67.32	48.38	64.45
webAI-ColVec1.1-4b (this model)	80.35	69.26	69.12	53.21	67.02	56.30	67.07	51.36	64.21
VultronRetriever Core	79.77	69.19	68.93	52.02	66.10	56.11	67.45	50.18	63.72
Nemotron ColEmbed VL 8B V2	79.29	69.82	67.29	51.54	66.32	56.03	67.19	50.84	63.54
webAI-ColVec1-4b	79.84	68.70	68.49	51.11	67.40	55.73	65.68	50.15	63.39
Tomoro ColQwen3 Embed 8B	75.35	68.41	65.08	49.10	63.98	54.41	66.36	50.13	61.60

The current MTEB leaderboard entries named webAI-ColVec1-4b and webAI-ColVec1-9b refer to the previous ColVec1 release, not these ColVec1.1 checkpoints.

💻 Usage

The processor provides the current retrieval API:

• process_images(images) prepares one or more document images.
• process_queries(texts) prepares one or more natural-language queries.
• score_retrieval(query_embeddings, document_embeddings) computes a MaxSim score matrix with shape (number_of_queries, number_of_documents).

Prerequisites and attention backends

The public ViDoRe V3 numbers are reproducible with this pinned FlashAttention 2 environment:

Python 3.12
PyTorch 2.9.0 + CUDA 12.8
Transformers 5.14.1
MTEB 2.18.5
Sentence Transformers 5.6.0
FlashAttention 2.8.3

The model also supports FlashAttention 3 on Hopper GPUs (H100/H200). See the Dao-AILab FlashAttention repository for FlashAttention 3 installation instructions, then select flash_attention_3 when loading in a compatible Hopper environment.

FlashAttention 2 and FlashAttention 3 both preserve bidirectional attention. FlashAttention 3 can improve throughput on Hopper GPUs, but the published scores use FlashAttention 2; changing kernels can produce small floating-point differences. Use FlashAttention 2 when reproducing the table.

Inference code

from io import BytesIO

import requests
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

MODEL_ID = "webAI-Official/webAI-ColVec1.1-4b"
DEVICE = "cuda:0" if torch.cuda.is_available() else "cpu"
ATTN_IMPLEMENTATION = (
    "flash_attention_2" if torch.cuda.is_available() else "sdpa"
)
# On an H100/H200 with FlashAttention 3 installed, use:
# ATTN_IMPLEMENTATION = "flash_attention_3"

processor = AutoProcessor.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    max_num_visual_tokens=1792,
)
model = AutoModel.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    attn_implementation=ATTN_IMPLEMENTATION,
    device_map=DEVICE,
).eval()

queries = [
    "Retrieve the city of Singapore",
    "Retrieve the city of Beijing",
]
document_urls = [
    "https://upload.wikimedia.org/wikipedia/commons/2/27/Singapore_skyline_2022.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/6/61/Beijing_skyline_at_night.JPG",
]


def load_image(url: str) -> Image.Image:
    response = requests.get(
        url,
        headers={"User-Agent": "Mozilla/5.0"},
        timeout=30,
    )
    response.raise_for_status()
    return Image.open(BytesIO(response.content)).convert("RGB")


device = next(model.parameters()).device
query_inputs = processor.process_queries(queries)
document_inputs = processor.process_images(
    [load_image(url) for url in document_urls]
)
query_inputs = {
    key: value.to(device) if isinstance(value, torch.Tensor) else value
    for key, value in query_inputs.items()
}
document_inputs = {
    key: value.to(device) if isinstance(value, torch.Tensor) else value
    for key, value in document_inputs.items()
}

with torch.inference_mode():
    query_batch = model(**query_inputs)
    document_batch = model(**document_inputs)

query_embeddings = [embedding.cpu() for embedding in query_batch]
document_embeddings = [embedding.cpu() for embedding in document_batch]
scores = processor.score_retrieval(
    query_embeddings,
    document_embeddings,
    output_dtype=torch.float32,
)

print(scores)
print("Best document per query:", scores.argmax(dim=1))

The processor loads the release's 1,792 visual-token budget by default. To reduce memory use, pass a lower max_num_visual_tokens value to AutoProcessor.from_pretrained; this changes document granularity and may change retrieval scores.

⚖️ Strengths and limitations

Strengths

• Strong public-task performance on ViDoRe V3.
• OCR-free retrieval over visually complex document pages.
• Fine-grained MaxSim matching with compact 640-dimensional token vectors.
• Multilingual training data and evaluation.
• FlashAttention 2 support across compatible CUDA GPUs and optimized FlashAttention 3 inference on Hopper GPUs.

Limitations

• Multi-vector indexes require more storage than single-vector embeddings.
• Scores can vary slightly between FlashAttention 2, FlashAttention 3, and SDPA.

License

Model weights are distributed under the webAI Non-Commercial License v1.0. See the repository's NOTICES.md for upstream attribution.

📚 Citation

@misc{webai_colvec1_1_4b,
  title  = {webAI-ColVec1.1-4b: A Bidirectional Multi-Vector Model for Visual Document Retrieval},
  author = {webAI},
  year   = {2026},
  url    = {https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b}
}