Instructions to use tomaarsen/mve-ModernBERT-base-token-pooling with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use tomaarsen/mve-ModernBERT-base-token-pooling with sentence-transformers:
from sentence_transformers import MultiVectorEncoder model = MultiVectorEncoder("tomaarsen/mve-ModernBERT-base-token-pooling") queries = ["Which planet is known as the Red Planet?"] documents = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) - Notebooks
- Google Colab
- Kaggle
mve-ModernBERT-base-token-pooling
This multi-vector retrieval model was trained from answerdotai/ModernBERT-base with a fresh 128-dimensional token projection. Its objective averages ordinary contrastive retrieval and retrieval after HierarchicalTokenPooling(pool_factor=4), teaching the encoder to work with approximately one quarter of the original document vectors. Queries remain unpooled.
Training used 120,000 MS MARCO triplets, seed 42, one epoch, batch size 256, encoder mini-batches of 64, a learning rate of 3e-5, and a training length limit of 300. Short queries are expanded to at least 32 token positions. The checkpoint supports other pooling settings at inference, and pooling must be selected explicitly when encoding documents.
This is a Multi-Vector Encoder model trained on the msmarco-bm25 dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.
Model Details
Model Description
- Model Type: Multi-Vector Encoder
- Maximum Sequence Length: 8192 tokens
- Output Dimensionality: 128 dimensions
- Similarity Function: MaxSim
- Supported Modality: Text
- Training Dataset:
- msmarco-bm25
Model Sources
- Documentation: Sentence Transformers Documentation
- Documentation: Multi-Vector Encoder Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Multi-Vector Encoders on Hugging Face
Full Model Architecture
MultiVectorEncoder(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_expansion': {'strategy': 'min', 'attend': False, 'token': None, 'length': 32}, 'architecture': 'ModernBertModel'})
(1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
(2): MultiVectorMask({'skiplist_words': [], 'skiplist_tasks': ['document'], 'keep_only_token_ids': None})
(3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)
Usage
Direct Usage (Sentence Transformers)
from sentence_transformers import MultiVectorEncoder
from sentence_transformers.multi_vector_encoder.modules import HierarchicalTokenPooling
model = MultiVectorEncoder("tomaarsen/mve-ModernBERT-base-token-pooling")
pooling = HierarchicalTokenPooling(pool_factor=4)
query_embeddings = model.encode_query(["How can I make a retrieval index smaller?"])
document_embeddings = model.encode_document(
[
"Token pooling merges similar token embeddings into fewer document vectors.",
"Paris is the capital of France.",
],
token_pooling=pooling,
)
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
Change pool_factor, use max_tokens for a per-document cap, or calibrate a distance_threshold for an average vector budget. Configure one of these controls at a time. The first token is protected by default and all output vectors are normalized.
Evaluation
NanoBEIR across pooling settings
Mean nDCG@10 (%) across all 13 NanoBEIR datasets with FP32 inference. Retention is the pooled mean divided by this model's unpooled mean. Vector savings use unchanged dimensionality and precision.
| Pooling at inference | Mean document vectors | nDCG@10 (%) | Score retained | Vectors saved |
|---|---|---|---|---|
| Unpooled | 193.22 | 52.39 | 100.0% | 0.0% |
max_tokens=64 |
54.76 | 50.89 | 97.1% | 71.7% |
max_tokens=32 |
29.58 | 50.12 | 95.7% | 84.7% |
max_tokens=16 |
15.60 | 49.26 | 94.0% | 91.9% |
max_tokens=8 |
7.99 | 46.61 | 89.0% | 95.9% |
max_tokens=4 |
4.00 | 43.00 | 82.1% | 97.9% |
pool_factor=2 |
96.86 | 51.74 | 98.8% | 49.9% |
pool_factor=3 |
64.74 | 51.45 | 98.2% | 66.5% |
pool_factor=4 |
48.68 | 51.23 | 97.8% | 74.8% |
pool_factor=5 |
39.04 | 50.28 | 96.0% | 79.8% |
pool_factor=6 |
32.62 | 49.85 | 95.2% | 83.1% |
| Adaptive threshold (54.76 mean vectors) | 54.76 | 51.24 | 97.8% | 71.7% |
| Adaptive threshold (29.58 mean vectors) | 29.58 | 50.96 | 97.3% | 84.7% |
| Adaptive threshold (15.60 mean vectors) | 15.60 | 49.63 | 94.7% | 91.9% |
Adaptive thresholds were calibrated separately for each corpus using unpooled document embeddings to match the average storage of fixed caps of 64, 32, and 16. These are exploratory results from one training seed, and the benchmark informed experiment development. Vector savings do not measure total search index size or end-to-end latency.
The metrics below were recorded by the training-time MS MARCO evaluators.
Metrics
Multi Vector Information Retrieval
- Dataset:
NanoMSMARCO - Evaluated with
MultiVectorInformationRetrievalEvaluator
| Metric | Value |
|---|---|
| maxsim_accuracy@1 | 0.3 |
| maxsim_accuracy@3 | 0.54 |
| maxsim_accuracy@5 | 0.62 |
| maxsim_accuracy@10 | 0.76 |
| maxsim_precision@1 | 0.3 |
| maxsim_precision@3 | 0.18 |
| maxsim_precision@5 | 0.124 |
| maxsim_precision@10 | 0.076 |
| maxsim_recall@1 | 0.3 |
| maxsim_recall@3 | 0.54 |
| maxsim_recall@5 | 0.62 |
| maxsim_recall@10 | 0.76 |
| maxsim_ndcg@10 | 0.5204 |
| maxsim_mrr@10 | 0.4456 |
| maxsim_map@100 | 0.4573 |
Multi Vector Information Retrieval
- Dataset:
NanoMSMARCO_pool_factor_4 - Evaluated with
MultiVectorInformationRetrievalEvaluatorwith these parameters:{ "token_pooling": { "type": "sentence_transformers.multi_vector_encoder.modules.token_pooling.HierarchicalTokenPooling", "config": { "pool_factor": 4, "max_tokens": null, "distance_threshold": null, "num_protected_tokens": 1, "normalize_embeddings": true, "tasks": [ "document" ] } } }
| Metric | Value |
|---|---|
| maxsim_accuracy@1 | 0.32 |
| maxsim_accuracy@3 | 0.52 |
| maxsim_accuracy@5 | 0.58 |
| maxsim_accuracy@10 | 0.74 |
| maxsim_precision@1 | 0.32 |
| maxsim_precision@3 | 0.1733 |
| maxsim_precision@5 | 0.116 |
| maxsim_precision@10 | 0.074 |
| maxsim_recall@1 | 0.32 |
| maxsim_recall@3 | 0.52 |
| maxsim_recall@5 | 0.58 |
| maxsim_recall@10 | 0.74 |
| maxsim_ndcg@10 | 0.5097 |
| maxsim_mrr@10 | 0.4392 |
| maxsim_map@100 | 0.4522 |
Training Details
Training Dataset
msmarco-bm25
- Dataset: msmarco-bm25
- Size: 120,000 training samples
- Columns:
query,positive, andnegative - Approximate statistics based on the first 100 samples:
query positive negative type string string string modality text text text details - min: 5 tokens
- mean: 9.2 tokens
- max: 17 tokens
- min: 31 tokens
- mean: 82.7 tokens
- max: 187 tokens
- min: 19 tokens
- mean: 83.45 tokens
- max: 184 tokens
- Samples:
query positive negative cost to install hardwood floor per roomHow Much Does Hardwood Floors Cost? Installing a standard solid-strip hardwood floor averages about $8 a square foot for materials, insulation and site finishing, and up to $12 a square foot for wide pine planks, or $1,150-$1,750 for a 12x12-foot room. Factory-finished wood starts at $8 a square foot installed, but generally runs $10-$14, or $1,140-$2,000 for a 12x12 space.Cost to Install a Hardwood Floor. 1 Share. 2 Hiring Local Professionals. 3 Sponsored Services. 4 Cost to Install a Hardwood Floor - Notes and General Information. 5 Cost to Install a Hardwood Floor - Unit Price References. 6 Hardwood Floor Installation - Average Cost Per Square Foot. 7 Help the homewyse Community.football colleges in georgiaFour-Year Georgia Universities or Colleges With Football Teams. There are 735 four-year colleges and universities that field football teams in the United States as of 2011. Six hundred thirty-nine belong to the National Collegiate Athletic Association's various divisions and 96 to the National Association of Intercollegiate Athletics.NCAA Division II (with football) Colleges in Georgia. If you're looking for Georgia NCAA Division II (with football) literacy rates, CALiteracy.org can help you find the schools with the most to offer. The information found here provides state literacy rates and information for NCAA Division II (with football) colleges across Georgia.what does glucosamine sulphate doGlucosamine sulfate is a chemical found in the human body. It is used by the body to produce a variety of other chemicals that are involved in building tendons, ligaments, cartilage, and the thick fluid that surrounds joints. Joints are cushioned by the fluid and cartilage that surround them.or example, the glucosamine sulfate that is put into dietary supplements is often harvested from the shells of shellfish. Glucosamine sulfate used in dietary supplements does not always come from natural sources. It can also be made in a laboratory.âGlucosamine sulphate, which is stabilized with sodium chloride or table salt, can contain as much as 30% sodium,â Keith says. âThis is a massive consideration for people who want to reduce their dietary intake of sodium (salt).â Glucosamine HCL and glucosamine sulphate 2KCL do not contain sodium chloride. - Loss:
MultiVectorTokenPoolingLosswith these parameters:{ "loss": "CachedMultiVectorMultipleNegativesRankingLoss", "token_poolings": [ null, "HierarchicalTokenPooling(pool_factor=4, max_tokens=None, distance_threshold=None, num_protected_tokens=1, normalize_embeddings=True, tasks=['document'])" ], "weights": [ 0.5, 0.5 ], "n_poolings_per_step": -1 }
Evaluation Dataset
msmarco-bm25
- Dataset: msmarco-bm25
- Size: 1,000 evaluation samples
- Columns:
query,positive, andnegative - Approximate statistics based on the first 100 samples:
query positive negative type string string string modality text text text details - min: 4 tokens
- mean: 9.0 tokens
- max: 15 tokens
- min: 35 tokens
- mean: 79.33 tokens
- max: 187 tokens
- min: 23 tokens
- mean: 79.28 tokens
- max: 187 tokens
- Samples:
query positive negative how much is a divorce in texasA divorce in Texas takes 61 days from the date of filing. You pay for only the first set of documents ($69.97) that you need to file with the court to start the divorce process. After 30 days you will pay for ($69.97) and receive the second half of the forms.o watch video. The average cost of divorce in the Texas courts is normally between $150 and $300, depending on the county you live in and the services you require.How to budget for a cost friendly divorce in Texas; Cost Cutting Tips for your Texas Divorce; Why do divorces cost so much in Texas? How Can I Get My Spouse to Pay My Attorney's Fees in a Texas Divorce? How am I going to Pay for My Texas Divorce? 8 Tips for Reducing the Cost of a Divorce in Texas; $300 Divorce Cost a Man $100,000 in Texashow much does the extended warranty for a volvo costI just bought one I purchased an Easy Care extended warranty from my local Volvo dealer about a week ago. I have a 2004 with 17,500 miles on it. I bought 4 yrs from the date I bought the warranty and a total of 100,000 total miles on the odometer with a zero deductible for $1832. The $100 deductible was between $1300 and $1400.Your carâs in the shop. You anxiously wait for the mechanic to call. How much is it going to cost this time? If this scenario sounds all too familiar, you might consider covering your vehicle with an extended car warranty. But how much does a car warranty cost?what type of procedure is used to determine hpvA colposcopy is a procedure that uses a colposcope (a lighted magnifying instrument) to examine the vulva, the vagina and cervix. A colposcopy is done after an abnormal Pap test or a positive HPV test suggests a precancerous condition of the cervix or cervical cancer.Periungual Warts. There are over 100 different strains of HPV identified. The specific strain of HPV that is present will determine what kind of warts will appear. Periungual warts are just one type of the wart family.1.here are over 100 different strains of HPV identified. The specific strain of HPV that is present will determine what kind of warts will appear. Periungual warts are just one type of the wart family. - Loss:
MultiVectorTokenPoolingLosswith these parameters:{ "loss": "CachedMultiVectorMultipleNegativesRankingLoss", "token_poolings": [ null, "HierarchicalTokenPooling(pool_factor=4, max_tokens=None, distance_threshold=None, num_protected_tokens=1, normalize_embeddings=True, tasks=['document'])" ], "weights": [ 0.5, 0.5 ], "n_poolings_per_step": -1 }
Training Hyperparameters
Non-Default Hyperparameters
per_device_train_batch_size: 256num_train_epochs: 1learning_rate: 3e-05warmup_steps: 0.05bf16: Trueper_device_eval_batch_size: 256data_seed: 42batch_sampler: no_duplicatesmax_length: 300
All Hyperparameters
Click to expand
per_device_train_batch_size: 256num_train_epochs: 1max_steps: -1learning_rate: 3e-05lr_scheduler_type: linearlr_scheduler_kwargs: Nonewarmup_steps: 0.05optim: adamw_torch_fusedoptim_args: Noneweight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08optim_target_modules: Nonegradient_accumulation_steps: 1average_tokens_across_devices: Truemax_grad_norm: 1.0label_smoothing_factor: 0.0bf16: Truefp16: Falsebf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonegradient_checkpointing: Falsegradient_checkpointing_kwargs: Nonetorch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneuse_liger_kernel: Falseliger_kernel_config: Noneuse_cache: Falseneftune_noise_alpha: Nonetorch_empty_cache_steps: Noneauto_find_batch_size: Falselog_on_each_node: Truelogging_nan_inf_filter: Trueinclude_num_input_tokens_seen: nolog_level: passivelog_level_replica: warningdisable_tqdm: Falseproject: huggingfacetrackio_space_id: trackioper_device_eval_batch_size: 256prediction_loss_only: Trueeval_on_start: Falseeval_do_concat_batches: Trueeval_use_gather_object: Falseeval_accumulation_steps: Noneinclude_for_metrics: []batch_eval_metrics: Falsesave_only_model: Falsesave_on_each_node: Falseenable_jit_checkpoint: Falsepush_to_hub: Falsehub_private_repo: Nonehub_model_id: Nonehub_strategy: every_savehub_always_push: Falsehub_revision: Noneload_best_model_at_end: Falseignore_data_skip: Falserestore_callback_states_from_checkpoint: Falsefull_determinism: Falseseed: 42data_seed: 42use_cpu: Falseaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}parallelism_config: Nonedataloader_drop_last: Falsedataloader_num_workers: 0dataloader_pin_memory: Truedataloader_persistent_workers: Falsedataloader_prefetch_factor: Noneremove_unused_columns: Truelabel_names: Nonetrain_sampling_strategy: randomlength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falseddp_backend: Noneddp_timeout: 1800fsdp: []fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}deepspeed: Nonedebug: []skip_memory_metrics: Truedo_predict: Falseresume_from_checkpoint: Nonewarmup_ratio: Nonelocal_rank: -1prompts: Nonebatch_sampler: no_duplicatesmulti_dataset_batch_sampler: proportionalrouter_mapping: {}learning_rate_mapping: {}max_length: 300
Training Logs
| Epoch | Step | Training Loss | Validation Loss | NanoMSMARCO_maxsim_ndcg@10 | NanoMSMARCO_pool_factor_4_maxsim_ndcg@10 |
|---|---|---|---|---|---|
| 0.0512 | 24 | 5.1169 | - | - | - |
| 0.1023 | 48 | 1.5040 | - | - | - |
| 0.1535 | 72 | 0.8463 | - | - | - |
| 0.2004 | 94 | - | 0.4934 | 0.4640 | 0.4658 |
| 0.2047 | 96 | 0.7092 | - | - | - |
| 0.2559 | 120 | 0.6407 | - | - | - |
| 0.3070 | 144 | 0.6077 | - | - | - |
| 0.3582 | 168 | 0.5839 | - | - | - |
| 0.4009 | 188 | - | 0.3914 | 0.5007 | 0.5104 |
| 0.4094 | 192 | 0.5508 | - | - | - |
| 0.4606 | 216 | 0.5369 | - | - | - |
| 0.5117 | 240 | 0.5393 | - | - | - |
| 0.5629 | 264 | 0.4945 | - | - | - |
| 0.6013 | 282 | - | 0.3720 | 0.4974 | 0.5141 |
| 0.6141 | 288 | 0.4989 | - | - | - |
| 0.6652 | 312 | 0.4688 | - | - | - |
| 0.7164 | 336 | 0.4795 | - | - | - |
| 0.7676 | 360 | 0.4878 | - | - | - |
| 0.8017 | 376 | - | 0.3626 | 0.5374 | 0.5160 |
| 0.8188 | 384 | 0.4654 | - | - | - |
| 0.8699 | 408 | 0.4819 | - | - | - |
| 0.9211 | 432 | 0.4880 | - | - | - |
| 0.9723 | 456 | 0.4486 | - | - | - |
| 1.0 | 469 | - | 0.3526 | 0.5204 | 0.5097 |
Training Time
- Training: 33.7 minutes
- Evaluation: 2.1 minutes
- Total: 35.8 minutes
Framework Versions
- Python: 3.12.3
- Sentence Transformers: 6.3.0.dev0
- Transformers: 5.4.0
- PyTorch: 2.11.0+cu128
- Accelerate: 1.13.0
- Datasets: 4.8.5
- Tokenizers: 0.22.2
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
MultiVectorTokenPoolingLoss
@inproceedings{josef2026learntopool,
title={Learn to Pool: Lightweight Fine-Tuning for Flexible Multi-Vector Compression},
author={Stefan Josef},
booktitle={1st Late Interaction Workshop (LIR) at ECIR 2026},
year={2026},
}
CachedMultiVectorMultipleNegativesRankingLoss
@misc{gao2021scaling,
title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
year={2021},
eprint={2101.06983},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
- Downloads last month
- 109
Model tree for tomaarsen/mve-ModernBERT-base-token-pooling
Base model
answerdotai/ModernBERT-basePapers for tomaarsen/mve-ModernBERT-base-token-pooling
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Article mentioning tomaarsen/mve-ModernBERT-base-token-pooling
Evaluation results
- Maxsim Accuracy@1 on NanoMSMARCOself-reported0.300
- Maxsim Accuracy@3 on NanoMSMARCOself-reported0.540
- Maxsim Accuracy@5 on NanoMSMARCOself-reported0.620
- Maxsim Accuracy@10 on NanoMSMARCOself-reported0.760
- Maxsim Precision@1 on NanoMSMARCOself-reported0.300
- Maxsim Precision@3 on NanoMSMARCOself-reported0.180
- Maxsim Precision@5 on NanoMSMARCOself-reported0.124
- Maxsim Precision@10 on NanoMSMARCOself-reported0.076
