R137 · Knowledge for agents › Retrieval over long documents

blink-multilingual-64M: a static embedding table that beats larger static models in 100+ languages

Read the original

embeddingsretrievalragopen-sourcebenchmarks

Source: Minish (Thomas van Dongen and Stephan Tulkens, the Model2Vec authors), release post 2026-11-18 and the model card on Hugging Face. MIT licence. All scores below are the authors' own, run with mteb 2.27.1.

What it is

blink-multilingual-64M is a static embedding model. There is no transformer at inference time: the whole model is one 250,000 × 256 table of token vectors (64M parameters, 256 MB in float32). To embed a text you tokenize it, look up each token's vector and take the mean. That makes encoding a memory lookup plus an average, orders of magnitude cheaper than running a transformer encoder, and it runs anywhere (Python via model2vec or Sentence Transformers, Rust via model2vec-rs).

It replaces the team's earlier potion-multilingual-128M at half the size, and follows an English-only blink-8M released a week earlier.

Headline numbers

MTEB Multilingual v2, all 131 tasks, every model compared at 256 dimensions:

ModelMean (task)RetrievalSTSBitext
blink-multilingual-64M51.6047.5267.4757.03
bekko-static-embedding-v1 (truncated to 256d)48.3747.6263.7237.30
potion-multilingual-128M47.6741.2261.1540.72
static-similarity-mrl-multilingual-v1 (256d)46.8839.4963.6748.50

The honest reading: a big jump over its predecessor and a clear lead on bitext and STS, but retrieval is a tie with bekko, not a win.

How it was built (six steps)

  1. Tokenizer. A 250k uncased WordPiece vocabulary that contains every letter of 57 scripts and splits long character runs, so nothing becomes [UNK]. A stock WordPiece lost 10% of Thai and 23% of Lao characters to [UNK]. 20k Chinese and 10k Japanese words are added as whole tokens.

  2. Initialization. Each token is embedded by bge-m3, reduced to 256 dimensions with PCA, and weighted by token frequency (SIF).

  3. Distillation (Tokenlearn). The table learns to reproduce bge-m3's sentence embeddings for 4M multilingual C4 texts, with a cosine loss.

  4. Contrastive training. Following the bekko recipe: MNRL at temperature 0.05 on 64M pairs (mMARCO, MIRACL, NLI, bitext and more), one data source per batch, hard negatives where available, trained natively at 256 dimensions.

  5. Weight averaging. Final weights are 0.8 × contrastive + 0.2 × distilled.

  6. Language codes. Contrastive training on translations erases which language a text is in. Each language gets a fixed random vector, and each token is nudged towards the languages it occurs in. Language-ID tasks jump (IndicLangClassification 26.3 → 64.9) for a 0.07-point average cost elsewhere. The model is still a plain table.

Architecture: where a static model fits

query / doc chunk -> tokenize (250k WordPiece) -> look up 256d rows -> mean pool + L2 norm
docs  -> vector index (256d)
query -> cosine top-k over index -> optional transformer re-ranker -> results

A static model is a candidate generator and a cheap similarity function, not a full replacement for a transformer encoder. The common shape is a two-stage pipeline: the table embeds the whole corpus and every query cheaply, then a transformer re-ranker or a stronger embedder scores the top few hundred hits.

Tradeoffs

Failure modes to watch

Review checklist

Connections

See R137 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.