mve-ModernBERT-base-token-pooling

This multi-vector retrieval model was trained from answerdotai/ModernBERT-base with a fresh 128-dimensional token projection. Its objective averages ordinary contrastive retrieval and retrieval after HierarchicalTokenPooling(pool_factor=4), teaching the encoder to work with approximately one quarter of the original document vectors. Queries remain unpooled.

Training used 120,000 MS MARCO triplets, seed 42, one epoch, batch size 256, encoder mini-batches of 64, a learning rate of 3e-5, and a training length limit of 300. Short queries are expanded to at least 32 token positions. The checkpoint supports other pooling settings at inference, and pooling must be selected explicitly when encoding documents.

This is a Multi-Vector Encoder model trained on the msmarco-bm25 dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

  • Model Type: Multi-Vector Encoder
  • Maximum Sequence Length: 8192 tokens
  • Output Dimensionality: 128 dimensions
  • Similarity Function: MaxSim
  • Supported Modality: Text
  • Training Dataset:
    • msmarco-bm25

Model Sources

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_expansion': {'strategy': 'min', 'attend': False, 'token': None, 'length': 32}, 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (2): MultiVectorMask({'skiplist_words': [], 'skiplist_tasks': ['document'], 'keep_only_token_ids': None})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

from sentence_transformers import MultiVectorEncoder
from sentence_transformers.multi_vector_encoder.modules import HierarchicalTokenPooling

model = MultiVectorEncoder("tomaarsen/mve-ModernBERT-base-token-pooling")
pooling = HierarchicalTokenPooling(pool_factor=4)
query_embeddings = model.encode_query(["How can I make a retrieval index smaller?"])
document_embeddings = model.encode_document(
    [
        "Token pooling merges similar token embeddings into fewer document vectors.",
        "Paris is the capital of France.",
    ],
    token_pooling=pooling,
)
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)

Change pool_factor, use max_tokens for a per-document cap, or calibrate a distance_threshold for an average vector budget. Configure one of these controls at a time. The first token is protected by default and all output vectors are normalized.

Evaluation

NanoBEIR across pooling settings

Mean nDCG@10 (%) across all 13 NanoBEIR datasets with FP32 inference. Retention is the pooled mean divided by this model's unpooled mean. Vector savings use unchanged dimensionality and precision.

Pooling at inference Mean document vectors nDCG@10 (%) Score retained Vectors saved
Unpooled 193.22 52.39 100.0% 0.0%
max_tokens=64 54.76 50.89 97.1% 71.7%
max_tokens=32 29.58 50.12 95.7% 84.7%
max_tokens=16 15.60 49.26 94.0% 91.9%
max_tokens=8 7.99 46.61 89.0% 95.9%
max_tokens=4 4.00 43.00 82.1% 97.9%
pool_factor=2 96.86 51.74 98.8% 49.9%
pool_factor=3 64.74 51.45 98.2% 66.5%
pool_factor=4 48.68 51.23 97.8% 74.8%
pool_factor=5 39.04 50.28 96.0% 79.8%
pool_factor=6 32.62 49.85 95.2% 83.1%
Adaptive threshold (54.76 mean vectors) 54.76 51.24 97.8% 71.7%
Adaptive threshold (29.58 mean vectors) 29.58 50.96 97.3% 84.7%
Adaptive threshold (15.60 mean vectors) 15.60 49.63 94.7% 91.9%

Proportional token pooling versus the matched ordinary-loss baseline

Adaptive thresholds were calibrated separately for each corpus using unpooled document embeddings to match the average storage of fixed caps of 64, 32, and 16. These are exploratory results from one training seed, and the benchmark informed experiment development. Vector savings do not measure total search index size or end-to-end latency.

The metrics below were recorded by the training-time MS MARCO evaluators.

Metrics

Multi Vector Information Retrieval

Metric Value
maxsim_accuracy@1 0.3
maxsim_accuracy@3 0.54
maxsim_accuracy@5 0.62
maxsim_accuracy@10 0.76
maxsim_precision@1 0.3
maxsim_precision@3 0.18
maxsim_precision@5 0.124
maxsim_precision@10 0.076
maxsim_recall@1 0.3
maxsim_recall@3 0.54
maxsim_recall@5 0.62
maxsim_recall@10 0.76
maxsim_ndcg@10 0.5204
maxsim_mrr@10 0.4456
maxsim_map@100 0.4573

Multi Vector Information Retrieval

  • Dataset: NanoMSMARCO_pool_factor_4
  • Evaluated with MultiVectorInformationRetrievalEvaluator with these parameters:
    {
        "token_pooling": {
            "type": "sentence_transformers.multi_vector_encoder.modules.token_pooling.HierarchicalTokenPooling",
            "config": {
                "pool_factor": 4,
                "max_tokens": null,
                "distance_threshold": null,
                "num_protected_tokens": 1,
                "normalize_embeddings": true,
                "tasks": [
                    "document"
                ]
            }
        }
    }
    
Metric Value
maxsim_accuracy@1 0.32
maxsim_accuracy@3 0.52
maxsim_accuracy@5 0.58
maxsim_accuracy@10 0.74
maxsim_precision@1 0.32
maxsim_precision@3 0.1733
maxsim_precision@5 0.116
maxsim_precision@10 0.074
maxsim_recall@1 0.32
maxsim_recall@3 0.52
maxsim_recall@5 0.58
maxsim_recall@10 0.74
maxsim_ndcg@10 0.5097
maxsim_mrr@10 0.4392
maxsim_map@100 0.4522

Training Details

Training Dataset

msmarco-bm25

  • Dataset: msmarco-bm25
  • Size: 120,000 training samples
  • Columns: query, positive, and negative
  • Approximate statistics based on the first 100 samples:
    query positive negative
    type string string string
    modality text text text
    details
    • min: 5 tokens
    • mean: 9.2 tokens
    • max: 17 tokens
    • min: 31 tokens
    • mean: 82.7 tokens
    • max: 187 tokens
    • min: 19 tokens
    • mean: 83.45 tokens
    • max: 184 tokens
  • Samples:
    query positive negative
    cost to install hardwood floor per room How Much Does Hardwood Floors Cost? Installing a standard solid-strip hardwood floor averages about $8 a square foot for materials, insulation and site finishing, and up to $12 a square foot for wide pine planks, or $1,150-$1,750 for a 12x12-foot room. Factory-finished wood starts at $8 a square foot installed, but generally runs $10-$14, or $1,140-$2,000 for a 12x12 space. Cost to Install a Hardwood Floor. 1 Share. 2 Hiring Local Professionals. 3 Sponsored Services. 4 Cost to Install a Hardwood Floor - Notes and General Information. 5 Cost to Install a Hardwood Floor - Unit Price References. 6 Hardwood Floor Installation - Average Cost Per Square Foot. 7 Help the homewyse Community.
    football colleges in georgia Four-Year Georgia Universities or Colleges With Football Teams. There are 735 four-year colleges and universities that field football teams in the United States as of 2011. Six hundred thirty-nine belong to the National Collegiate Athletic Association's various divisions and 96 to the National Association of Intercollegiate Athletics. NCAA Division II (with football) Colleges in Georgia. If you're looking for Georgia NCAA Division II (with football) literacy rates, CALiteracy.org can help you find the schools with the most to offer. The information found here provides state literacy rates and information for NCAA Division II (with football) colleges across Georgia.
    what does glucosamine sulphate do Glucosamine sulfate is a chemical found in the human body. It is used by the body to produce a variety of other chemicals that are involved in building tendons, ligaments, cartilage, and the thick fluid that surrounds joints. Joints are cushioned by the fluid and cartilage that surround them.or example, the glucosamine sulfate that is put into dietary supplements is often harvested from the shells of shellfish. Glucosamine sulfate used in dietary supplements does not always come from natural sources. It can also be made in a laboratory. “Glucosamine sulphate, which is stabilized with sodium chloride or table salt, can contain as much as 30% sodium,” Keith says. “This is a massive consideration for people who want to reduce their dietary intake of sodium (salt).” Glucosamine HCL and glucosamine sulphate 2KCL do not contain sodium chloride.
  • Loss: MultiVectorTokenPoolingLoss with these parameters:
    {
        "loss": "CachedMultiVectorMultipleNegativesRankingLoss",
        "token_poolings": [
            null,
            "HierarchicalTokenPooling(pool_factor=4, max_tokens=None, distance_threshold=None, num_protected_tokens=1, normalize_embeddings=True, tasks=['document'])"
        ],
        "weights": [
            0.5,
            0.5
        ],
        "n_poolings_per_step": -1
    }
    

Evaluation Dataset

msmarco-bm25

  • Dataset: msmarco-bm25
  • Size: 1,000 evaluation samples
  • Columns: query, positive, and negative
  • Approximate statistics based on the first 100 samples:
    query positive negative
    type string string string
    modality text text text
    details
    • min: 4 tokens
    • mean: 9.0 tokens
    • max: 15 tokens
    • min: 35 tokens
    • mean: 79.33 tokens
    • max: 187 tokens
    • min: 23 tokens
    • mean: 79.28 tokens
    • max: 187 tokens
  • Samples:
    query positive negative
    how much is a divorce in texas A divorce in Texas takes 61 days from the date of filing. You pay for only the first set of documents ($69.97) that you need to file with the court to start the divorce process. After 30 days you will pay for ($69.97) and receive the second half of the forms.o watch video. The average cost of divorce in the Texas courts is normally between $150 and $300, depending on the county you live in and the services you require. How to budget for a cost friendly divorce in Texas; Cost Cutting Tips for your Texas Divorce; Why do divorces cost so much in Texas? How Can I Get My Spouse to Pay My Attorney's Fees in a Texas Divorce? How am I going to Pay for My Texas Divorce? 8 Tips for Reducing the Cost of a Divorce in Texas; $300 Divorce Cost a Man $100,000 in Texas
    how much does the extended warranty for a volvo cost I just bought one I purchased an Easy Care extended warranty from my local Volvo dealer about a week ago. I have a 2004 with 17,500 miles on it. I bought 4 yrs from the date I bought the warranty and a total of 100,000 total miles on the odometer with a zero deductible for $1832. The $100 deductible was between $1300 and $1400. Your car’s in the shop. You anxiously wait for the mechanic to call. How much is it going to cost this time? If this scenario sounds all too familiar, you might consider covering your vehicle with an extended car warranty. But how much does a car warranty cost?
    what type of procedure is used to determine hpv A colposcopy is a procedure that uses a colposcope (a lighted magnifying instrument) to examine the vulva, the vagina and cervix. A colposcopy is done after an abnormal Pap test or a positive HPV test suggests a precancerous condition of the cervix or cervical cancer. Periungual Warts. There are over 100 different strains of HPV identified. The specific strain of HPV that is present will determine what kind of warts will appear. Periungual warts are just one type of the wart family.1.here are over 100 different strains of HPV identified. The specific strain of HPV that is present will determine what kind of warts will appear. Periungual warts are just one type of the wart family.
  • Loss: MultiVectorTokenPoolingLoss with these parameters:
    {
        "loss": "CachedMultiVectorMultipleNegativesRankingLoss",
        "token_poolings": [
            null,
            "HierarchicalTokenPooling(pool_factor=4, max_tokens=None, distance_threshold=None, num_protected_tokens=1, normalize_embeddings=True, tasks=['document'])"
        ],
        "weights": [
            0.5,
            0.5
        ],
        "n_poolings_per_step": -1
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 256
  • num_train_epochs: 1
  • learning_rate: 3e-05
  • warmup_steps: 0.05
  • bf16: True
  • per_device_eval_batch_size: 256
  • data_seed: 42
  • batch_sampler: no_duplicates
  • max_length: 300

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 256
  • num_train_epochs: 1
  • max_steps: -1
  • learning_rate: 3e-05
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: None
  • warmup_steps: 0.05
  • optim: adamw_torch_fused
  • optim_args: None
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 1
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: False
  • project: huggingface
  • trackio_space_id: trackio
  • per_device_eval_batch_size: 256
  • prediction_loss_only: True
  • eval_on_start: False
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: False
  • hub_private_repo: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 42
  • data_seed: 42
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: []
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • warmup_ratio: None
  • local_rank: -1
  • prompts: None
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}
  • max_length: 300

Training Logs

Epoch Step Training Loss Validation Loss NanoMSMARCO_maxsim_ndcg@10 NanoMSMARCO_pool_factor_4_maxsim_ndcg@10
0.0512 24 5.1169 - - -
0.1023 48 1.5040 - - -
0.1535 72 0.8463 - - -
0.2004 94 - 0.4934 0.4640 0.4658
0.2047 96 0.7092 - - -
0.2559 120 0.6407 - - -
0.3070 144 0.6077 - - -
0.3582 168 0.5839 - - -
0.4009 188 - 0.3914 0.5007 0.5104
0.4094 192 0.5508 - - -
0.4606 216 0.5369 - - -
0.5117 240 0.5393 - - -
0.5629 264 0.4945 - - -
0.6013 282 - 0.3720 0.4974 0.5141
0.6141 288 0.4989 - - -
0.6652 312 0.4688 - - -
0.7164 336 0.4795 - - -
0.7676 360 0.4878 - - -
0.8017 376 - 0.3626 0.5374 0.5160
0.8188 384 0.4654 - - -
0.8699 408 0.4819 - - -
0.9211 432 0.4880 - - -
0.9723 456 0.4486 - - -
1.0 469 - 0.3526 0.5204 0.5097

Training Time

  • Training: 33.7 minutes
  • Evaluation: 2.1 minutes
  • Total: 35.8 minutes

Framework Versions

  • Python: 3.12.3
  • Sentence Transformers: 6.3.0.dev0
  • Transformers: 5.4.0
  • PyTorch: 2.11.0+cu128
  • Accelerate: 1.13.0
  • Datasets: 4.8.5
  • Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultiVectorTokenPoolingLoss

@inproceedings{josef2026learntopool,
    title={Learn to Pool: Lightweight Fine-Tuning for Flexible Multi-Vector Compression},
    author={Stefan Josef},
    booktitle={1st Late Interaction Workshop (LIR) at ECIR 2026},
    year={2026},
}

CachedMultiVectorMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
Downloads last month
109
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for tomaarsen/mve-ModernBERT-base-token-pooling

Finetuned
(1,605)
this model

Papers for tomaarsen/mve-ModernBERT-base-token-pooling

Article mentioning tomaarsen/mve-ModernBERT-base-token-pooling

Evaluation results