tuskanny/scidocs_modern_colbert

Dataset

SCIDOCS, GTE-ModernColBERT

3 commits

1 linked in READMEs

updated Sep 23, 2026

See the code

README

SCIDOCS, GTE-ModernColBERT

Token-level (late-interaction) embeddings of the BEIR SCIDOCS corpus and queries, encoded with GTE-ModernColBERT, in the TACHIOM multivector format.

Source

BEIR SCIDOCS, test (its only split) split. Corpus, queries and qrels were read from the official BEIR files via ir_datasets (beir/scidocs); PyLate only did the encoding
25,657 documents, 1,000 queries, 29,928 qrels
Text given to the encoder for each document: title + " " + text (BEIR title and body joined by a space). The text itself is not included, only its vectors
Row order follows the BEIR corpus and query files; row i of doc_ids.npy / queries_ids.npy identifies row i of doclens.npy / queries.npy

Encoding

Model: lightonai/GTE-ModernColBERT-v1 @ 25f6f7bb8237b7ae25ae1d9b805ce17c0d1cc639
Library: PyLate 1.6.0, CPU
Document length cap: 300 tokens (model default)
Query length: 48 tokens (model default)
Query expansion: no (model default)
The model defaults come from config_sentence_transformers.json in the model repository. We did not override any of them
Documents: punctuation tokens are dropped (PyLate skiplist), as are padding tokens
Vectors: 128-d, L2-normalized

Statistics


Token vectors (N)	4,819,277
Avg vectors per document	187.8 (max 300)
Vectors per query	variable, 6 to 48 (no query expansion), zero-padded to 48
Avg vectors per query	17.5

Files

File	dtype	Shape	Content
`documents.npy`	float16 (`<f2`)	`[4819277, 128]`	All document vectors, concatenated document by document
`doclens.npy`	int32	`[25657]`	Vectors per document; `sum == N`
`token_ids.npy`	uint32	`[4819277]`	Input token id of each row of `documents.npy`
`doc_ids.npy`	string	`[25657]`	BEIR doc id of each document
`queries.npy`	float32	`[1000, 48, 128]`	Query vectors, zero-padded at the end
`query_lens.npy`	int32	`[1000]`	True number of vectors per query
`queries_ids.npy`	string	`[1000]`	BEIR query id of each query
`qrels.test.tsv`	TREC	29928 lines	`qid \t 0 \t docid \t relevance`
`groundtruth/gt_top100.tsv`	TSV	100000 lines	Exhaustive top-100: `query_idx \t doc_idx \t rank \t score` (0-based positions)
`groundtruth/gt_ids.npy`	int32	`[1000, 100]`	Same, as doc positions
`groundtruth/gt_scores.npy`	float32	`[1000, 100]`	Same, MaxSim scores

Zero padding does not change any score: a zero query vector adds 0 to the MaxSim of every document.

Exhaustive-search effectiveness

Exact MaxSim over the full collection (vectorium compute_groundtruth_multivec). These are the reference numbers for approximate search on this data.

nDCG@10	R@100
0.1949	0.4407

beir

colbert

late-interaction

multivector

scidocs

Similar repos

tuskanny/scidocs_colbertv2

Dataset

SCIDOCS, ColBERTv2

tuskanny/nfcorpus_modern_colbert

Dataset

NFCorpus, GTE-ModernColBERT

tuskanny/scidocs_lateon

Dataset

SCIDOCS, LateOn

tuskanny/fiqa_modern_colbert

Dataset

FiQA-2018, GTE-ModernColBERT

robro612/scidocs_mlateon

Dataset

scidocs_mlateon

tuskanny/nfcorpus_colbertv2

Dataset

NFCorpus, ColBERTv2

robro612/scidocs_answerai_colbert_small

Dataset

scidocs_answerai_colbert_small

robro612/scifact_lateon

Dataset

scifact_lateon

Linked in READMEs

TusKANNy/awesome-multivector-retrieval

An extensive and commented list of resources on Late-Interaction Multivector Retrieval.

README badge

tuskanny/scidocs_modern_colbert

Dataset

SCIDOCS, GTE-ModernColBERT

3 commits

1 linked in READMEs

updated Sep 23, 2026

See the code

README

SCIDOCS, GTE-ModernColBERT

Token-level (late-interaction) embeddings of the BEIR SCIDOCS corpus and queries, encoded with GTE-ModernColBERT, in the TACHIOM multivector format.

Source

BEIR SCIDOCS, test (its only split) split. Corpus, queries and qrels were read from the official BEIR files via ir_datasets (beir/scidocs); PyLate only did the encoding
25,657 documents, 1,000 queries, 29,928 qrels
Text given to the encoder for each document: title + " " + text (BEIR title and body joined by a space). The text itself is not included, only its vectors
Row order follows the BEIR corpus and query files; row i of doc_ids.npy / queries_ids.npy identifies row i of doclens.npy / queries.npy

Encoding

Model: lightonai/GTE-ModernColBERT-v1 @ 25f6f7bb8237b7ae25ae1d9b805ce17c0d1cc639
Library: PyLate 1.6.0, CPU
Document length cap: 300 tokens (model default)
Query length: 48 tokens (model default)
Query expansion: no (model default)
The model defaults come from config_sentence_transformers.json in the model repository. We did not override any of them
Documents: punctuation tokens are dropped (PyLate skiplist), as are padding tokens
Vectors: 128-d, L2-normalized

Statistics


Token vectors (N)	4,819,277
Avg vectors per document	187.8 (max 300)
Vectors per query	variable, 6 to 48 (no query expansion), zero-padded to 48
Avg vectors per query	17.5

Files

File	dtype	Shape	Content
`documents.npy`	float16 (`<f2`)	`[4819277, 128]`	All document vectors, concatenated document by document
`doclens.npy`	int32	`[25657]`	Vectors per document; `sum == N`
`token_ids.npy`	uint32	`[4819277]`	Input token id of each row of `documents.npy`
`doc_ids.npy`	string	`[25657]`	BEIR doc id of each document
`queries.npy`	float32	`[1000, 48, 128]`	Query vectors, zero-padded at the end
`query_lens.npy`	int32	`[1000]`	True number of vectors per query
`queries_ids.npy`	string	`[1000]`	BEIR query id of each query
`qrels.test.tsv`	TREC	29928 lines	`qid \t 0 \t docid \t relevance`
`groundtruth/gt_top100.tsv`	TSV	100000 lines	Exhaustive top-100: `query_idx \t doc_idx \t rank \t score` (0-based positions)
`groundtruth/gt_ids.npy`	int32	`[1000, 100]`	Same, as doc positions
`groundtruth/gt_scores.npy`	float32	`[1000, 100]`	Same, MaxSim scores

Zero padding does not change any score: a zero query vector adds 0 to the MaxSim of every document.

Exhaustive-search effectiveness

Exact MaxSim over the full collection (vectorium compute_groundtruth_multivec). These are the reference numbers for approximate search on this data.