ModernBERT-base-en-v1

This is a Multi-Vector Encoder model created by a 50/50 linear merge of chungimungi/GLInt and lightonai/LateOn-regularized, using the MultiVectorEncoder.merge method from the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

The merged model scores above the average of its two parents on 9 of the 13 NanoBEIR datasets, including 5 where it beats both. It achieves a mean nDCG@10 of 0.7061, compared with 0.6996 for GLInt and 0.6979 for LateOn-regularized, using the same inference settings for all three models.

It took a whopping 6 lines of code to merge the models and set the document length, and it took about 12 seconds to execute. See Model Creation for the full generation script. The merging settings were not based on any evaluation results.

Model Details

Model Description

• Model Type: Multi-Vector Encoder
• Base models: GLInt and LateOn-regularized
• Maximum Sequence Length: 8192 tokens
  • Maximum Query Length: 32 tokens
  • Maximum Document Length: 8192 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: MaxSim
• Supported Modality: Text
• Language: English
• License: Apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Multi-Vector Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_length': 32, 'document_length': 8192, 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 1536, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings', 'use_residual': True})
  (2): Dense({'in_features': 1536, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings', 'use_residual': True})
  (3): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (4): MultiVectorMask({'skiplist_words': ['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~'], 'skiplist_tasks': ['document'], 'keep_only_token_ids': None})
  (5): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("multi-vector-encoder/ModernBERT-base-en-v1")
queries = [
    'Which planet is known as the Red Planet?',
]
documents = [
    "Venus is often called Earth's twin because of its similar size and proximity.",
    'Mars, known for its reddish appearance, is often referred to as the Red Planet.',
    'Saturn, famous for its rings, is sometimes mistaken for the Red Planet.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (12, 128) (18, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[11.4814, 11.6525, 11.6232]])

Evaluation

Metrics

NanoBEIR

• Dataset: NanoBEIR-en
• Evaluated with MultiVectorNanoBEIREvaluator on all 13 datasets.
• Query length: 32 tokens. Document length: 8192 tokens (expanded from 300 for GLInt and LateOn-regularized).
• Metric: nDCG@10. The mean is unweighted across datasets.

The parent models were evaluated with the same prompts and length limits.

Dataset	ModernBERT-base-en-v1	GLInt	LateOn-regularized
msmarco	0.736155	0.728477	0.707793
nq	0.818848	0.818032	0.782732
fiqa2018	0.640317	0.625973	0.625546
nfcorpus	0.393756	0.399912	0.391630
scidocs	0.435285	0.435472	0.444901
arguana	0.588440	0.569680	0.561172
dbpedia	0.703868	0.709176	0.678083
quoraretrieval	0.990127	0.983948	0.986797
touche2020	0.692082	0.696390	0.686696
climatefever	0.484580	0.418405	0.521409
fever	0.966278	0.969763	0.966278
hotpotqa	0.930661	0.934504	0.917731
scifact	0.799062	0.804524	0.802028
mean	0.706112	0.699558	0.697907

Model Creation

The backbone and all three projection layers were merged in fp32 with equal weights from GLInt and LateOn-regularized, without any training.

This is all it took:

# pip install "sentence-transformers[merge]"
from sentence_transformers import MultiVectorEncoder

# Load the GLInt and LateOn-regularized models,
# matching the latter's special token config
glint = MultiVectorEncoder(
    "chungimungi/GLInt",
    config_kwargs={
        "bos_token_id": 50281,
        "eos_token_id": 50282,
        "pad_token_id": 50283,
    },
    processor_kwargs={"model_max_length": 8192, "max_length": 8192},
)
lateon = MultiVectorEncoder("lightonai/LateOn-regularized")

# Merge using simple 50/50 linear
model = MultiVectorEncoder.merge(
    [glint, lateon],
    method="linear",
    parameters={"weight": [0.5, 0.5]},
)
# Increase the document length from 300 to 8192 tokens
model[0].document_length = 8192
# And save it for reuse
model.save_pretrained("ModernBERT-base-en-v1")

Framework Versions

• Python: 3.12.3
• Sentence Transformers: 6.2.0
• Transformers: 5.4.0
• PyTorch: 2.11.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.8.5
• Tokenizers: 0.22.2

Citation

BibTeX

LateOn Regularization

@misc{chaffin2026lateonregularization,
  title={Party is over: regularizing ColBERT models to fix efficient ANN methods},
  author={Chaffin, Antoine},
  year={2026},
  howpublished={\url{https://huggingface.co/blog/lightonai/lateon-regularization}},
}

GLInt

@misc{aarush2026glint,
  title={GLInt: Geometry-Matched Hard Negatives for Late-Interaction Retrieval},
  author={Aarush},
  year={2026},
  howpublished={\url{https://huggingface.co/blog/chungimungi/glint}},
}

Mergekit

@inproceedings{goddard-etal-2024-arcees,
    title = "Arcee{'}s {M}erge{K}it: A Toolkit for Merging Large Language Models",
    author = "Goddard, Charles and Siriwardhana, Shamane and Ehghaghi, Malikeh and Meyers, Luke and Karpukhin, Vladimir and Benedict, Brian and McQuade, Mark and Solawetz, Jacob",
    editor = "Dernoncourt, Franck and Preo{\c{t}}iuc-Pietro, Daniel and Shimorina, Anastasia",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2024",
    address = "Miami, Florida, US",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-industry.36",
    doi = "10.18653/v1/2024.emnlp-industry.36",
    pages = "477--485",
}