arXiv Complete Corpus

A snapshot of arXiv's metadata, version history, submission files and rendered documents. It covers 3,148,796 papers and includes file contents, paths, sizes and SHA-256 digests. Metadata comes from arXiv's OAI-PMH arXivRaw interface; files come from the GCS mirror, S3 source archives and direct PDF fetches.

This release holds a PDF for 99.47% of papers and 99.54% of versions reported with a non-zero submission size. It is a one-off snapshot; coverage gaps are documented below.

Start here

Start with the 26 MB sample:

from datasets import load_dataset

ds = load_dataset("secemp9/arxiv-complete", "sample", split="train")
paper = ds[0]
print(paper["paper_id"], paper["title"], len(paper["text"]))

Choose paper_text for one resolved TeX string per paper, latex for the individual TeX-related files, source for complete submission packages, or pdf for rendered documents. Use metadata, versions and files to plan a selection before fetching content.

The main limits to account for are:

• paper_text retains LaTeX syntax, comments and macros. It includes withdrawal stubs and some incorrectly selected templates; it needs filtering before use as a training corpus.
• Source packages are unversioned. A paper's newest held PDF can be older than the newest version arXiv reports.
• Large binary rows and row groups require substantial memory. Early exit from datasets streams also failed on the versions tested for this release.
• The html family is indexed in files but has no content config.
• Licences vary by paper. The compilation's CC0 dedication does not apply to the papers; see LICENSE.

Coverage · Reading data · paper_text · source · latex and duplication · Schema · Provenance and verification · File quality

Configs

Parquet sizes are compressed bytes on disk, in decimal SI units (1 TB = 1,000,000,000,000 bytes). All nine configs total 16,076,056,758,128 bytes, or 16.08 TB. Elsewhere, raw content bytes means sum(size) over the relevant files, before Parquet compression. For example, pdf contains 10.37 TB of raw content in 8.65 TB of Parquet; source contains 9.52 TB in 6.51 TB. Sizes and percentages below identify which basis they use.

papers is count(DISTINCT paper_id). All counts cover every shard of the named config. No config contains a paper absent from metadata, and paper_id is never null. sample is drawn from paper_text to demonstrate the schema. Use the full configs for coverage counts.

SELECT count(*) AS rows, count(DISTINCT paper_id) AS papers
FROM 'hf://datasets/secemp9/arxiv-complete/ps/*.parquet';
-- 2,237,839 rows; 1,318,462 papers

The author-supplied configs overlap: latex is a subset of source, and paper_text is assembled from its .tex and .ltx files plus formatted bibliographies. pdf and ps contain rendered documents and, in some cases, additional assets. PostScript represents rendered pages; LaTeX is source code.

Download one config

An unfiltered snapshot_download() attempts to download all 16.08 TB. Use allow_patterns to select a config:

from huggingface_hub import snapshot_download

snapshot_download(
    "secemp9/arxiv-complete",
    repo_type="dataset",
    allow_patterns="paper_text/*",
)

The three index configs total 4,345,782,824 Parquet bytes (4.35 GB). files describes 22,571,129,586,192 raw content bytes, including the indexed but unpublished html family. Remote SQL queries can project columns and prune row groups without downloading whole configs. They still transfer the metadata and column data needed by the query.

Snapshot and coverage

There are 353 papers dated 1991 and 284,162 dated 2025. The largest primary categories are cs.CV (5.0%), hep-ph (4.6%), cs.LG (4.5%), quant-ph (4.3%), hep-th (3.6%), astro-ph (3.0%) and cs.CL (2.8%). For 67 papers, first_version_date preserves an original preprint date that predates the month in paper_id.

Version coverage

Of the 4,996,775 versions arXiv reports with non-zero submission size, 4,973,594 (99.54%) have a PDF here. The table uses retrievable as shorthand for a non-zero reported size; it does not guarantee a live download exists.

SELECT count(*) AS versions,
       count(*) FILTER (WHERE is_metadata_only) AS metadata_only,
       count(*) FILTER (WHERE NOT is_metadata_only) AS retrievable,
       count(*) FILTER (WHERE NOT is_metadata_only AND has_pdf) AS with_pdf,
       count(*) FILTER (WHERE NOT is_metadata_only
                         AND (has_pdf OR has_ps OR has_html)) AS with_any,
       count(*) FILTER (WHERE NOT is_metadata_only AND NOT has_pdf
                         AND NOT has_ps AND NOT has_html) AS real_gaps,
       count(*) FILTER (WHERE NOT has_pdf AND NOT has_ps AND NOT has_html)
           AS no_artifact
FROM 'hf://datasets/secemp9/arxiv-complete/versions/*.parquet';

has_pdf alone is true for 4,973,947 versions, including 353 marked metadata-only. That is 98.87% of all 5,030,612 reported versions. The 99.54% figure excludes metadata-only rows from both numerator and denominator. At paper level, 3,131,959 of 3,148,796 papers (99.47%) have at least one PDF; 3,133,383 (99.51%) have a PDF, PostScript or HTML artifact indexed.

is_metadata_only is true exactly when arXiv's <size> is 0kb, parsed as reported_size_kb = 0. Neither column is null. These versions affect 29,221 papers and are the newest version of 16,490 of them. They often correspond to withdrawals. The flag records arXiv's reported size; 509 metadata-only versions still have an artifact here (330 PDF and PostScript, 156 HTML, 23 PDF alone). Consequently, the 54,519 versions with no artifact consist of 21,191 non-zero-size gaps and 33,328 metadata-only versions.

On 2026-09-13, live arXivRaw records for 40 papers covering 122 versions agreed with every size flag. A separate check of 20 metadata-only PDF URLs returned 20 HTTP 404s; all 10 held-version controls returned PDFs. These checks establish agreement only for the versions tested.

Most non-zero-size gaps fall near the snapshot boundary:

Daily PDF coverage was 97–99% through 2026-08-12, 81.7% on August 13 and 11–15% thereafter. All 20 sampled snapshot-edge URLs returned PDFs on 2026-09-13. Of the 2,674 older gaps, 1,405 date to 1991–1995; 15 sampled URLs from that group returned 404. The two groups need different treatment.

versions indexes what arXiv reports; files indexes what the build held. Every true has_pdf, has_ps and has_html flag has a matching files row. The PDF and PostScript pair counts agree exactly at 4,973,947 and 1,953,094. In the other direction, files contains 36 HTML (paper, version) pairs across 21 papers that arXiv's OAI record does not report, including 1412.7030 v111 and 1611.09139 v70. See the HTML omission.

PDF provenance and the gap-filling pass

The 6,475 web_scrape PDFs were fetched from versioned arxiv.org URLs on 2026-08-30. They use the mirror's path layout and are scattered across 1,218 of 1,886 PDF shards. stored_as, stored_size and stored_sha256 are null for both provenances; source distinguishes them. Every fetched row's digest agrees with versions.pdf_sha256.

Fetched PDFs range from 31,284 bytes to 129.8 MB (2503.01938v1), with a median of 847 KB versus 636 KB for gcs; two exceed 64 MB. All 2,652 held PDFs for versions dated August 14 onward came from this pass. Across the corpus, 2,146 papers and 6,266 version pairs have no rendered artifact except one of these fetched PDFs. A gcs row may hold an older rendering; web_scrape records what arxiv.org served on the fetch date.

SELECT source, count(*), count(DISTINCT paper_id), sum(size)
FROM 'hf://datasets/secemp9/arxiv-complete/files/*.parquet'
WHERE kind = 'pdf'
GROUP BY source;

Requests and outcomes

The pass targeted 10,085 version pairs missing a PDF on 2026-08-30. Each was requested from /pdf/, the path arXiv's robots.txt explicitly allows, from a single host behind a global rate limiter that doubles its interval on any throttle signal. Only the first 289 went at the fifteen-second Crawl-delay that file declares; the remaining 9,796 went at a 1.20-second global interval — 0.83 requests a second, 12.5× the rate arXiv publishes. arXiv signalled throttling zero times in 10,224 requests, but the faster rate was our choice and it is recorded here rather than left out. Every non-success was requested a second time. The final tally, one row per target:

The zero recovery rate for 1991–1995 applies to the missing-PDF request list. The dataset already holds PDFs for 30,898 of that era's 32,341 retrievable versions (95.5%); for 1993 alone, it holds 5,985 of 6,530 (91.7%).

Remaining gaps

All 1,640 pre-2026 gaps were requested without success. Of the 3,610 failed targets, only 1,642 remain gaps under the no-artifact definition: 1,967 hold PostScript or HTML (1,809 HTML), and 1808.02949 v1 holds nine PDFs under names such as v1.1.pdf, although its standard version URL failed.

Recheck on 2026-09-13

The recheck covered 144 distinct version URLs, requested at fifteen-second intervals. The rows below are disjoint; two URLs that appeared in two strata were counted once and returned the same response on both requests.

All 98 previously requested URLs repeated their earlier failure: 63 returned 404, 18 returned 500 and 17 returned an empty 200. All 38 previously unasked URLs and all 8 controls returned PDFs, with no transport errors or throttling. The failures persisted between the two checks. Untested URLs and later availability remain unverified.

The 19,549 gaps never requested by the build are all dated 2026 and account for 92.2% of the gap set. This query selects them together with two 2026 versions that returned HTTP 500:

SELECT paper_id, version, version_date, arxiv_pdf_url
FROM 'hf://datasets/secemp9/arxiv-complete/versions/*.parquet'
WHERE NOT is_metadata_only
  AND NOT has_pdf AND NOT has_ps AND NOT has_html
  AND version_date >= DATE '2026-01-01';
-- 19,551 rows

At fifteen seconds per request, a pass takes about 81 hours before retries. If all 19,549 unasked versions yielded PDFs, retrievable-version PDF coverage would rise to about 99.93%. The sample supports attempting these fetches, but does not establish that outcome. Check arXiv's current access rules before starting a new pass.

Papers with no available text route

An inventory check found 13,912 papers (0.442%) with no published text route: 13,898 have no files and 14 have files but no usable text route. The other 3,134,884 (99.558%) meet at least one route below. The table counts matching records without assessing their text quality: a paper_text row can still be a stub, and a PDF can be blank or defective.

The first six routes can be reproduced from the indexes:

WITH reach AS (
  SELECT paper_id,
         max(kind = 'pdf') AS in_pdf,
         max(kind = 'ps') AS in_ps,
         max(kind = 'source' AND lower(path) LIKE '%.pdf') AS src_pdf,
         max(kind = 'source' AND regexp_matches(lower(path),
             '\.(ps|eps|epsi|epsf|pstex)$')) AS src_ps,
         max(kind = 'source' AND regexp_matches(lower(path),
             '\.(html?|xhtml|shtml)$')) AS src_html
  FROM 'hf://datasets/secemp9/arxiv-complete/files/*.parquet'
  GROUP BY paper_id
)
SELECT m.paper_id
FROM 'hf://datasets/secemp9/arxiv-complete/metadata/*.parquet' m
LEFT JOIN reach r USING (paper_id)
LEFT JOIN (
  SELECT paper_id
  FROM 'hf://datasets/secemp9/arxiv-complete/paper_text/*.parquet'
) t ON t.paper_id = m.paper_id
WHERE t.paper_id IS NULL
  AND NOT coalesce(r.in_pdf OR r.in_ps OR r.src_pdf OR r.src_ps
                   OR r.src_html, false);
-- 13,913 rows, before the extensionless-document check

Adding latex or .dvi files as routes changes this result by zero. Indexed html rows are excluded because their content config was omitted. Among the 2,864 papers with source but no PDF or PostScript row, 2,485 have paper_text; the remaining 379 hold 6,596 source files (118,117,724 bytes). All those files were read, classified from their bytes and checked against their recorded lengths and hashes, with zero mismatches. Of the 379 papers, 363 ship HTML in source. Filename errors did not falsely add a text route for any of this residual set.

The 2,864 source-only papers hold 10,930 source rows and 450,068,220 raw bytes, with a median of one file. Their 3,624 version rows include 1,781 with HTML and 1,457 non-zero-size gaps. They largely predate the snapshot edge: 1,427 first appeared in 1991–1995 and the newest on 2026-07-27. Their leading categories are hep-ph (473), hep-th (369) and cs.LO (222).

The seventh route is gr-qc/9408005: its entire paper, The Black Hole History in Tamed Vacuum, is 65,618 bytes of Phyzzx TeX in the extensionless file BHTAME, opening %%%%Phyzzx file and closing \end. A later census found extensionless TeX documents in 1,663 papers without paper_text; 1,659 already have PDFs and three have PostScript, leaving only this paper to add to the table. See extensionless files.

All 13,898 fileless papers first appeared in 2026, earliest on March 18; 12,965 appeared on August 14 or later. Their 14,788 version rows carry no artifact and only 16 are metadata-only. This is principally snapshot lag. Nine have withdraw in their comments, versus 8,713 papers corpus-wide, so the evidence does not support describing the whole group as withdrawals.

The fourteen remaining papers are:

All 31 files in the fifteen-paper residual set before the BHTAME correction were read and re-hashed, with zero size or digest mismatches. Twelve of the fourteen have withdrawal or removal remnants. Ten of the eleven notes start with %auto-ignore; gr-qc/9410009 instead starts %Paper: gr-qc/9410009. The three redirected works are available under their replacement identifiers, with 56,518, 37,298 and 30,500 characters of paper_text, respectively. The other two cases are the figure-only submission and the unpublished HTML proceedings. This count records which papers lack a published text route.

On 2026-09-14, the newest PDF URL for each of the fifteen candidates, plus v1 for the three with multiple versions, returned 18 HTTP 404s. Four controls returned PDFs. The check does not establish that no copy exists elsewhere.

This availability count exceeds the 3,133,383 papers with an indexed arXiv render: 1,502 additional papers are reachable through their submission files, while 2608.02072 has an indexed render whose bytes are not published. Thus 3,133,383 + 1,502 − 1 = 3,134,884.

Reading data

All SQL examples use DuckDB.

Grouping files by paper

latex, source, pdf and ps have one row per file. Rows are sorted by paper_id, contiguous for each paper and never split across shards. The audit checked ordering and boundaries across all 295,391 row groups in the four configs' 3,686 shards.

Each maximum belongs to one paper. In source, 697,611 papers (22.35%) have one row, 42,535 exceed a hundred and 358 exceed a thousand. The last group contains 638,026 rows and 14.70 GB of raw content. Single-row shares are 38.20% in latex, 60.81% in pdf and 63.19% in ps.

To assemble each paper's LaTeX files:

import itertools
from datasets import load_dataset

ds = load_dataset(
    "secemp9/arxiv-complete", "latex", split="train", streaming=True
)
# Each contiguous paper_id group contains the paper's LaTeX files.
for pid, rows in itertools.groupby(ds, key=lambda r: r["paper_id"]):
    files = {r["path"]: r["content"] for r in rows}
    print(pid, len(files), sum(map(len, files.values())))

This loop runs to exhaustion. Read the early-exit caveat before shortening it with break.

The dictionary holds an entire paper's raw content. File count alone does not predict its memory cost:

With pyarrow 25.0.1, assembling local rows into that dictionary produced:

The bare-interpreter baseline was 66 MiB. 1309.2485 is the Model Checking Contest 2013 report; 16,213 rows are PDF-named files totalling 158.2 MB, with a median size of 7,550 bytes. Its row groups require 67.5 MB compressed and 195.6 MB uncompressed. The 32-row coupled-cluster submission 2412.11029 has 28 ancillary files; its four groups require 158.9 MB compressed and 3,399.3 MB uncompressed.

In source and latex, grouping collects files from one unversioned package. In pdf, it mostly collects revisions: 4,973,956 rows represent 4,973,947 version pairs. Only two pairs have multiple rows (1808.02949 v1 has nine; 1611.09018 v3 has a PDF and a text file). The 187-row maximum is therefore 187 revisions, and mean rows and versions per paper both round to 1.5881.

ps combines revisions and assets: 2,237,839 rows represent 1,953,094 pairs, of which 1,882,090 have one row and 71,004 have more. That is 284,745 rows above one per pair, with means of 1.481 versions and 1.697 rows per paper. 0912.3460 has two versions, with 768 and 765 objects. The distinction from its 293,617 asset count is explained under PostScript.

Row groups and memory

The readers measured for this release load substantially more than one row when reading Parquet content. Compressed row-group size estimates transfer cost; uncompressed size is a useful starting point for memory budgeting. Peak RSS can be several times larger because of decoding, prefetching and conversion to Python objects. Small batch_size values did not remove that cost in the tested readers.

These measurements cover all 301,750 row groups across all 3,745 shards. Sizes are decimal MB; rows/RG is the packer's cap and commonest value. Groups can be shorter at shard boundaries or for unusually large payloads.

Repacking the retained large groups would require rewriting 9.49 TB of pdf, ps and latex Parquet, estimated at 20–24 hours plus upload. In pdf, 38.23% of groups exceed 250 MB compressed. PostScript compresses well but still has a 203.9 MB median uncompressed group.

With datasets 5.0.1 and pyarrow 25.0.1, fetching one row measured:

The remote transfer comprised 11.6 MB of setup traffic and 114.4 MB of content from two prefetched groups (56.6 MB and 57.7 MB). Shard 00696's first group is 977.4 MB uncompressed over 187 rows; the result is not a measured worst case for the whole config.

Project out content when you only need an index. Non-content columns total 241 MB compressed for pdf, 108 MB for ps, 551 MB for latex and 2.36 GB for source. In paper_text, columns other than text total 1.24 GB. For example, this reads only metadata columns from a PDF shard and can stop after one batch:

from huggingface_hub import HfFileSystem
import pyarrow.parquet as pq

fs = HfFileSystem()
path = "datasets/secemp9/arxiv-complete/pdf/train-00000-of-01886.parquet"
with fs.open(path, "rb") as handle:
    pf = pq.ParquetFile(handle)
    batches = pf.iter_batches(
        batch_size=64, columns=["paper_id", "path", "sha256"]
    )
    print(next(batches).to_pydict())

On a tested local PDF shard, this projection read 9.4 kB and peaked at 66 MiB RSS against a 60 MiB baseline; including content read 516 MB and peaked at 1,155 MiB. datasets also accepts columns=["paper_id", "version", "path", "size", "sha256"].

By comparison, changing datasets to batch_size=1 reduced one measurement from 1,993 to 1,836 MiB on a 623.8 MB group. Arrow's iter_batches(batch_size=1) still read 516 MB and peaked at 1,155 MiB in the other test. A mid-file read of one 0.78 MB PDF required 609.5 MB of reads and 2,007 MiB RSS from a 706.8 MB group.

Allow roughly 2–4 GB per PDF or PostScript reader for ordinary groups, with more for the exceptional rows below. For selected papers, locate and query their shards instead of streaming the whole config. For file categories such as anc/, shard selection is much less effective; use a reader that can evaluate a small column before fetching content.

Streaming early exit

The release audit found interpreter-shutdown hangs or aborts when a datasets iterator was abandoned. The tested environment was datasets 5.0.1, pyarrow 25.0.1 and Python 3.12.3. Results after taking one row over the Hub and ending the script were:

The failure occurs after the row is delivered. It can leave a job running or return a failed exit status; it is not evidence that the delivered row was corrupted. The abort included PyGILState_Release during finalization. A local 288 MB files shard also hung in 4 of 4 tests, excluding the network as a necessary cause. An all-config process aborted in 11 of 11 tests.

An abandoned Parquet generator was observed finalizing after pyarrow had been torn down, raising an ArrowInvalid attribute error. One hung process had thirteen threads in futex_wait_queue, after 3 seconds of useful CPU work and 282 seconds without further CPU progress.

Related upstream reports include Arrow #45214, closed as a duplicate of #49942, with fix #49943. Those reports do not override the release's measurements: the audit reproduced failures on 25.0.1 and also tested 24.0.0. Results should be rechecked when changing library versions.

For partial reads, use DuckDB or the direct Arrow example above. Forcing the older-version cleanup branch in datasets still aborted in 6 of 6 tests. os._exit bypassed shutdown, but also bypasses buffer flushing and cleanup; an unconditional zero-status exit can hide application errors. It is not a general replacement for normal program termination.

Largest rows

A reader returning content as one Python bytes value must fit that value in memory, regardless of batch size. Repacking can reduce neighbouring data in a row group but cannot reduce the size of the file itself.

The table covers all file sizes in files and UTF-8 encoded string sizes in paper_text. Units are decimal.

Using 64 MiB (67,108,864 bytes) instead changes the counts to 338 in source, 1,177 in pdf, 422 in ps and one in latex.

All six rows exceeding 1 GB were read back, hashed and inspected. Five exceed 1 GiB:

The first two are ancillary text data. The rendered documents have valid format headers and end markers. latex's largest file is 2508.04390's 72,454,841-byte anthology.bib; paper_text's largest row is 2109.06332, 45,888,100 UTF-8 bytes and 45,888,090 characters.

None of the 1,937 indexed files above 64 MiB matches the hash of a zero-filled buffer of its own size. The largest NUL-filled file is 33,496,382 bytes, below this range.

The largest PDF sits alone in row group 9 of pdf/train-00740-of-01886.parquet: 1,113,321,202 bytes uncompressed, 13,365,541 compressed. On pyarrow 25.0.1:

The Python result cost about four times the row's 1,061.8 MiB. The PDF's image streams compress well in Parquet (83:1 for this group), so its small network transfer does not imply a small memory requirement. The largest row accounts for 99.99993% of its group in pdf, 99.4% in ps and 98.2% in source. In latex, the 72.5 MB bibliography shares a 231.3 MB group with 1,999 other files.

Use the index to identify large values before fetching them:

SELECT kind, paper_id, version, path, size
FROM 'hf://datasets/secemp9/arxiv-complete/files/*.parquet'
WHERE size > 64000000
ORDER BY size DESC;
-- 2,166 rows, 256.9 GB of raw content

The six files above 1 GB occupy five shards: pdf/train-00740, ps/train-00100, ps/train-00115, source/train-00482 and source/train-01155. Filtering after a streaming row arrives does not avoid its read cost. If exact snapshot bytes are unnecessary, arxiv_pdf_url may provide a smaller current rendering; 2206.04656v2 was 21,742,481 bytes there on 2026-09-13. See PDF rendering changes.

Latest held versions

pdf and ps retain the revisions the build obtained. Superseded PDF rows account for 3,798,892,308,546 of 10,371,932,999,324 raw content bytes (36.6%) and 1,841,997 of 4,973,956 rows (37.03%). Those shares do not measure compressed Parquet savings, because row groups mix revisions.

The 268,733,475-byte versions config supplies three flags:

For 20,652 papers, the newest held PDF is older than the newest version in the metadata: 0.659% of the 3,131,959 PDF-bearing papers, or 0.656% of all papers. This comparison is within the snapshot; it is not a live check of arXiv's current revision. Filtering only on is_latest_version would lose these papers from a latest-held-PDF view.

from datasets import load_dataset

v = load_dataset("secemp9/arxiv-complete", "versions", split="train")
latest = {(r["paper_id"], r["version"]) for r in v if r["is_latest_pdf"]}
pdf = load_dataset(
    "secemp9/arxiv-complete", "pdf", split="train", streaming=True
)
# Membership selects the latest held version; all files of that pair are kept.
for row in pdf:
    if (row["paper_id"], row["version"]) in latest:
        print(row["paper_id"], row["version"], row["path"], row["size"])

The filter runs after transfer, so streaming still reads every revision. Project out content if you only need the inventory, then retrieve selected shards. The flags live in versions to avoid rewriting the 9.33 TB of PDF and PostScript Parquet. For a partial run, use the alternatives described in streaming early exit.

paper_text

paper_text provides one assembled TeX string for each of 2,856,227 papers. It retains preambles, comments and macros; it is not plain prose extracted from rendered pages. The 991-row sample has the same schema.

The remaining 292,569 metadata papers divide into 265,111 with source but no .tex or .ltx, 13,560 with no source package, and 13,898 with no files. The first group includes PDF-only submissions, encrypted .cry files and 1,663 papers with extensionless TeX. Accounting for those documents leaves 290,906 papers outside both paper_text and this additional TeX route; the three set differences above are unchanged.

Text size and token estimate

The totals come from all 50 shards. There are 174,595,098 non-ASCII characters (0.0707%); the remainder use one UTF-8 byte each. Encoded byte length is distinct from Python object memory and from the 70 GB compressed Parquet download. Raw UTF-8 content compresses by about 3.5:1.

The estimate of 78–81 billion tokens uses cl100k_base, a sample of 2,586 papers and observed ratios of 3.05–3.14 characters per token. It is not a full token count. Using the measured character total gives 78.6–80.9 billion; other tokenizers need their own ratio:

estimated tokens = 246,780,951,228 / measured characters per token

The original estimate used byte counts with a characters-per-token ratio. This added 0.0945%, below the range's precision. A twelve-core benchmark of encode_ordinary_batch(num_threads=12) in tiktoken processed 41.0 million characters per second, suggesting about 1.7 hours for tokenization alone, plus input time. A full pass was not run.

Quantiles are discrete order statistics. There are 2,694 papers over a million characters, 800,271 at or above a hundred thousand, and 16,581 below a thousand.

from datasets import load_dataset

ds = load_dataset(
    "secemp9/arxiv-complete", "paper_text", split="train", streaming=True
)
# This visits the full config; length is measured in Unicode characters.
for paper in ds:
    print(paper["paper_id"], paper["title"], len(paper["text"]))

Row groups are substantially smaller than in the binary configs, but the early-exit issue also affected paper_text on the tested library versions.

Assembly and omitted files

The builder follows \input and \include depth-first, in command order, from a selected main file. It uses path-order concatenation when it cannot find a main file or the expansion covers less than half the TeX bytes.

Unused files are excluded from resolved rows. They can be drafts, alternate sections or the actual paper when the main-file selection is wrong. resolution records the assembly method. Even a resolved row may omit the paper or include unrelated text. Bibliography databases (.bib) are excluded; formatted .bbl files are appended if the include graph did not already incorporate them.

The builder leaves 465,530 TeX files out of 185,982 papers: 6.65% of its 6,998,107 input TeX files. Only resolved rows have non-zero n_unused_files; 35.71% of those rows omit at least one. Among affected papers, the median omitted count is 1, p75 is 2, p90 is 5 and p99 is 18. The maximum is 1,323 for 2306.09110, which uses 1,361 of its 2,684 files.

For every row, n_tex_files = n_files_used + n_unused_files. Totals are 6,998,107 = 6,532,577 + 465,530, matching latex's 6,997,048 tex files plus 1,059 ltx files. The 1,341,591 .bbl rows are outside these counts. Every skipped input remains available in latex:

SELECT paper_id, path, size, sha256, content
FROM 'hf://datasets/secemp9/arxiv-complete/latex/*.parquet'
WHERE paper_id = '2306.09110' AND ext IN ('tex', 'ltx');
-- 2,684 rows

main_file names the selected entry point, but the used-file list is not published. Recovering the exact omitted set requires resolving the include graph again from the individual files.

Repeated text and incorrect main-file selection

There are 2,844,004 distinct text_sha256 values. Only 320 texts repeat, across 12,543 rows. The largest group is the 11,369 identical bare %auto-ignore stubs. Reading the other 319 groups found:

Of the 108 long-text groups, twelve have at least three copies, accounting for 107 rows; all twelve are publisher templates or macro packages. The clearest case is the 77,099-character AIP aipguide.tex, selected as main_file in 44 resolved rows. All 44 packages also contain the real paper among three to five TeX files, such as 1411.4279's paper.tex, 1106.5937's heavy11.tex and 1009.1634's ms.tex. Those papers remain available in latex even though paper_text.text contains the guide.

Other repeated guides include IOP (14, 5 and 3 copies), MNRAS (7), World Scientific (7), jpconf (7 and 5), Springer EDITOR.TEX (5), ws-procs975x65 (4), AMS-TeX (3) and IAU (3). The 96 two-copy groups include latex.ltx, mtexsis.tex, eplain.tex, amstex.tex, lineno.sty and ACM, LLNCS, AASTeX, CVPR, ICCV, NeurIPS, AISTATS and MICCAI templates. Two matching texts alone are not proof of a resolver error.

The largest non-stub group by row count is a 134-character \includepdf wrapper shared by 45 papers, from 1608.07343 through 1807.08563. Duplicate hashes are useful review candidates, but removing all duplicates does not establish that the remaining rows contain the intended papers.

Short rows and document markers

Of the 16,581 rows under 1,000 characters (0.58%), 13,235 contain %auto-ignore and 3,078 contain \includepdf; these two sets are disjoint. The remaining 268 were not exhaustively classified. Samples contain unmarked withdrawal notes and empty preambles. There are 14,449 rows under 200 characters and 11,431 at 20 characters or fewer.

The wrappers have a median length of 192 characters. In 3,076 cases the embedded PDF is present in source, totalling 5,733,788,446 raw bytes. Their paper text must be extracted from that PDF. Filtering at length(text) >= 1000 removes this short-row population. Longer rows still require quality checks.

An earlier revision of this page reported 2,677 wrappers, 2,675 with PDFs, 4.95 GB and 669 residual rows. Its intermediate file truncated text at 400 characters, hiding \includepdf in 401 rows. The counts above use the full published strings.

A marker-only filter is unreliable. %auto-ignore appears somewhere in 14,751 rows, including 1,516 at least 1,000 characters long; 1,449 contain it after the first character. Of the 13,302 rows that start with it, 154 exceed 10,000 characters and contain ordinary papers. The share of rows at 20 characters or fewer ranges from 0.02% to 0.94% across shards. See withdrawal and removal records for the complete set of recorded forms.

Document markers also need context:

• 182,539 rows (6.39%) lack \documentclass.
• 37,401 (1.31%) lack both \documentclass and \begin{document}.
• 150,623 of the first group use the older \documentstyle; 145,138 still contain \begin{document}.
• Counting both class-declaration spellings leaves 31,887 rows (1.12%) with no declaration and no \begin{document}. Of these, 13,337 are marker-bearing or short stubs, 7,757 are complete plain-TeX or AMS-TeX documents, and 10,793 mix fragments, further plain TeX, HTML indexes and non-text payloads.

The last share ranges from 0.29% to 9.92% across shards. These statistics were measured over the full config; one shard is not representative.

Decoding

The builder joins the selected files with newlines, then decodes the whole paper once. It tries UTF-8 and falls back to latin-1 after any decoding error. text_encoding records the branch used for the assembled paper. Original file encodings are not recorded. One invalid byte can therefore change the decoding of the whole document.

The fallback rate ranges from 0.48% in shard 00044 to 17.92% in shard 00003. Its rows contain 83,056,146 non-ASCII characters, 0.519% of their text but 47.6% of the corpus's non-ASCII characters. ASCII is unchanged by either decoding branch.

Encoding the fallback strings back to latin-1 recovers the bytes presented to the decoder. Across those bytes, 6,253,320 non-ASCII bytes (7.53%) belong to valid UTF-8 multibyte sequences; 167,125 rows (87.3%) contain no such sequence. That rules out UTF-8 as a complete decoding of those rows, but does not establish latin-1 as the correct original encoding.

The audit grouped all fallback rows with heuristic byte tests:

Overlapping tests were assigned in this order: predominantly UTF-8, four-byte run, C1 byte, Mac OS Roman, remainder. Without that precedence, 69,668 rows contain a C1 byte, 52,113 a Windows-1252 smart-punctuation byte, 21,506 a four-byte run and 18,170 the Mac OS Roman pattern. These tests provide clues about source encodings; they do not identify every original codepage.

Among the 167,125 rows with no valid UTF-8 multibyte sequence, 56,730 still contain C1 bytes. Examples place byte 96 where Windows-1252 uses an en dash: 0709.2497 has pp. 141<96>--147, and 0807.0432 has reaction<96>diffusion. Samples of the CJK-shaped group contain Shift-JIS, EUC-JP, EUC-KR and GBK, often in comments. The full group was not decoded file by file.

In the 9,805 predominantly UTF-8 rows, 707,201 valid UTF-8 characters became 1,520,231 mojibake characters: 0.118% of those papers' text and 0.00062% of the corpus. For example, 1306.5649 joins 18 TeX files into 722,877 bytes. One invalid ED byte at offset 378,295 causes fallback, misdecoding 4,648 otherwise valid non-ASCII characters. Use latex to decode the individual source files with a different policy.

Replacement characters already in the source

The UTF-8 branch is a strict decode, yet 16,076 of its rows contain 633,175 U+FFFD replacement characters. The builder does not use replacement decoding, and none of the 191,366 latin-1 rows contains U+FFFD. These characters were already encoded as EF BF BD in the input files. Source checks, including 1909.05796 and 1211.6716, found the same bytes in latex; reading that config cannot recover the replaced characters.

The distribution is concentrated: 6,157 rows have one, 5,150 have two or three, and 3,082 have four to ten. Just 33 rows contain over a thousand each and account for 486,098 occurrences (76.8%). Across the UTF-8 branch, the total is 0.000274% of its characters. Runs of at least two occur in 3,271 rows, and runs of at least three in 1,287.

The sparse cases appear in accented names and punctuation. The heavy cases contain binary material embedded in TeX: 0802.2024 has 111,696 replacements in 375,686 characters from a PDF inside PrionBimodalHAL.tex; 1005.0686 and 1811.06523 also contain PDF material, while 1705.10234 and 1210.1903 contain JPEG material. Four papers (1803.00172, 1901.01315, 2010.11203, 2101.10652) share a 287,489-byte gzip payload named JHEP3.cls.tex, each with 47,494 replacements. Long rows can therefore still contain non-text data.

Licence populations

license, title, abstract and primary_category are copied from metadata. The licence values agree on every joined row, but the populations differ:

Of the 292,569 papers outside paper_text, 27,933 (9.55%) have no recorded licence. Papers with a recorded licence average 1.625 versions versus 1.434 for those without, explaining the lower version-level NULL share. The arXiv non-exclusive licence occurs in 1,734,444 paper_text rows (60.72%), compared with 60.42% of metadata papers. Each version repeats the paper-level value; a per-version licence history was not harvested.

Source content

source contains every extracted submission file, including figures, TeX, code, supplementary data and build artifacts. Files were not filtered for relevance. Most raw bytes belong to files named as figures or images:

The exact raw total is 9,518,477,174,660 bytes. The compressed download is 6.51 TB. Categories use the lowercased final dotted alphanumeric component of path, with disjoint extension sets:

The largest entries in the last category are xml (11.34 GB), no alphanumeric extension (7.77 GB), then ttf, fig, emf, ldf, pgf and ai. A suffix such as .fdb_latexmk falls in that category because it contains an underscore.

SELECT count(*), sum(size)
FROM 'hf://datasets/secemp9/arxiv-complete/files/*.parquet'
WHERE kind = 'source'
  AND lower(regexp_extract(path, '\.([A-Za-z0-9]+)$', 1))
      IN ('pdf','png','eps','jpg','jpeg','gif','ps','svg','tif','tiff',
          'bmp','pstex','epsf');
-- 31,911,285 files; 8,426,469,965,364 raw bytes

kind = 'source' and source = 's3_src_extracted' select the same rows. These categories are based on filenames. Actual formats and uses can differ: 14,246,890 PDF-named files account for 3,614,260,349,571 raw bytes (38.0% of source, about 43% of the figure category). Some are entire papers: 257,333 rows belong to packages without .tex or .ltx, totalling 428,146,685,889 bytes. See PDF-only source.

GIFs are assigned only to figures: 21,495 files, 5,375,567,265 bytes. In a 150-file random sample, 137 held one image, 11 were animated (12–500 frames), one could not be parsed to the end and one was an xv thumbnail. Three animations were under anc/. Moving every GIF to video would change the shares to 0.58% video/audio and 88.47% figures. An earlier table counted GIFs in both categories, incorrectly inflating video/audio to 29,852 files; the current categories are disjoint.

The video count uses the eleven container extensions above, excluding audio and GIF: 8,000 files, 49,873,942,150 raw bytes, 2,646 papers. Of these, 5,796 are .mp4 (29.94 GB). Restricting to .mp4, .mpg, .mpeg, .avi and .mov gives 7,786 files and 48.44 GB. Audio adds 212 .mp3 and 331 .wav files. Video formats have not been exhaustively identified; the ancillary sample below found substantial misnaming among .mpg/.mpeg.

Other retained material includes 8,473 files in .git/ directories across 58 papers (242,871,243 raw bytes), 3,185 Jupyter notebooks, and 1,141 __pycache__/ files across 136 papers (7,871,857 bytes). Of the latter, 1,140 end in .pyc. Match __pycache__ with contains(path, '__pycache__'): SQL LIKE '%__pycache__%' also matches entropycache_parameter_ablation.tex, producing an incorrect 1,142.

Extensionless files

Under the alphanumeric-suffix definition, 273,784 files have no extension: 7,774,078,264 raw bytes, 93,544 papers and 201,422 distinct blobs. Of these, 120,421 have no dot (README, Makefile, LICENSE, latexmkrc); 153,363 have a non-alphanumeric final component, including 54,383 .pdf_tex, 25,117 .pstex_t, 20,275 .fdb_latexmk and 20,096 .svn-base files.

All 273,784 files were read from the published Parquet, covering 79,532 source row groups, and checked against their lengths and digests with zero mismatches. Classification used signatures and TeX structure:

Plain text and TeX fragments make up 243,231 files (88.8%); complete TeX documents are 1.68%. The latter matter for paper_text coverage: among 2,174 extensionless files in 1,730 papers without a paper_text row, 1,675 files in 1,663 papers are complete documents, totalling 89,848,258 bytes. They comprise 1,612 LaTeX-shaped files and 63 plain-TeX files using forms such as \input amstex, \input harvmac, %%%%Phyzzx or \magnification, with \bye or \end endings.

The other 499 files comprise 232 texts without document structure, 194 PostScript or binary files, 67 under a kilobyte and six class/package files. The 1,663 document-bearing papers date from 1994-02-18 to 2025-04-14; 761 predate 2000, with physics categories leading. Of them, 1,659 have a PDF, three have PostScript and one has neither (gr-qc/9408005, the BHTAME case). Another file brings 299 of them into latex, but none into paper_text.

Ancillary files under anc/

arXiv's ancillary-file convention places supplementary data, code, images and other material in an anc directory at the submission root. This release preserves that directory in source. Most of its content is available in no other config.

Root-level anc/ contains 149,573 files across 11,485 papers and 83,668,861,836 raw bytes: 0.32% of source files, 0.879% of source bytes and 0.37% of source-bearing papers. Select it with starts_with(path, 'anc/'). contains(path, 'anc/') returns 150,597 because it also matches unrelated names such as MontBlanc/ and BruitBlanc/ (172 files, 16 papers). Another 852 files in 159 papers are under nested /anc/ directories; including them gives 150,425 files, 11,493 papers and 84,287,349,819 bytes.

The same disjoint extension sets used above give:

This directory holds 77.18% of source's video/audio bytes, 57.85% of its data/code bytes and 45.54% of its archive bytes. Its 4,790 video files account for 38.64 GB. There are 2,003 extension spellings; leading ones are:

The PDF-named files span 4,918 papers; 3,097 have supp in their path (2,844 papers), and 359 are named si.pdf. Common basenames include supplementary.pdf (259 papers) and supplement.pdf (241). There are 1,607 README files across 1,167 papers.

A stratified sample of 2,111 ancillary files across 1,237 row groups was read and hashed. Of 1,785 with extensions having a canonical format, 26 (1.46%) disagreed with their names. The other 326 used ambiguous names such as .dat, .db, .sav, .fig, .mx or no extension. This sample covers different formats from the general extension sample and is not directly comparable to its 0.21% mismatch rate.

Sixteen of 61 sampled .mpg/.mpeg files were other containers (26%): twelve ISO-BMFF and four AVI. For example, 1010.3081's anc/video1.mpg is MP4, and 1310.0925's anc/TheShockAndTheTurbulence_HighResolution.mpg is AVI. A sampled .avi from 2511.23380 is ASF/WMV. Of 31 .mat files, 23 were MATLAB level-5, one level-4 and seven plain text. One .npy was a 40-byte PAX record.

The remaining tested canonical formats matched, including 76 PDFs, 46 PNGs, 46 MP4s, 32 JPGs, 31 GIFs and 31 EPS files; 30 each of 7z, xz, FITS, HDF5, ROOT, XLSX, DOCX, PS and JPEG; 22 WMV, 20 WebM, 19 MP3, 16 WAV and 14 RAR. Signature checks need format-specific handling: six sampled tar files used v7 headers without ustar, and three QuickTime files began with mdat rather than ftyp.

Ancillary files by submission year and examples

Years below refer to the paper's first version. Its anc/ files may have arrived in a later revision. The 2026 row stops at August 27 (225,912 metadata papers), so it is a partial year.

first submitted	papers with ancillary files	files	raw bytes	share of that year's papers
2001	1	7	34,362	0.003%
2007	1	42	163,309	0.002%
2008	6	20	255,631,426	0.010%
2009	9	373	108,385,874	0.014%
2010	126	488	5,872,107,307	0.179%
2011	194	1,109	4,996,001,989	0.253%
2012	238	1,837	6,031,402,882	0.282%
2013	297	2,403	5,777,034,123	0.320%
2014	258	4,433	1,057,429,650	0.264%
2015	315	11,150	978,023,374	0.300%
2016	468	13,021	1,099,617,101	0.413%
2017	656	4,850	1,492,020,821	0.530%
2018	796	5,837	2,347,398,319	0.567%
2019	831	9,092	3,001,434,810	0.533%
2020	1,003	11,394	4,346,720,690	0.563%
2021	980	9,062	8,476,760,223	0.540%
2022	963	11,913	6,676,153,375	0.518%
2023	945	13,904	7,005,635,311	0.452%
2024	1,058	12,167	10,041,369,339	0.434%
2025	1,170	14,153	8,375,229,823	0.412%
2026	1,170	22,318	5,730,307,728	0.518%

0706.0489's 42 files arrived with its 2010 revision. The oldest example, cs/0106057, has one version dated 2001-06-28 and seven Perl files for an OAI-PMH harvester. Large file counts can come from a few submissions: two papers contribute 5,561 and 3,560 of the 13,021 files in the 2016 row.

Examples read and re-hashed from published Parquet include:

• 2412.11029, anc/CCSDTQ_os_12_r123t123_eqns_permu-R_pre.txt: 1,568,567,496 bytes of coupled-cluster coefficients; its 28 ancillary files total 3.33 GB.
• 2103.14690, anc/NHS_catalog_final.cds: 1,141,065,026 bytes of the NEWFIRM HETDEX Survey catalogue in fixed-width CDS format.
• 2104.02716, anc/arxiv_programs/20_eqs_on_phi: 680,955,946 bytes of Mathematica equations.
• 2507.08247, anc/realizable-3terminal.txt: 293,661,418 bytes of combinatorial-game values for Hex positions.
• 1803.05205, anc/combinatorial3spheres9vertices_with_proofs.txt: 194,667,981 bytes of enumerations and coordinate realizations.
• 2301.11363, anc/AmpSolid: 219,324,297 bytes of symbolic amplitudes; 2308.11686, anc/Spin_1_runGrid_45.dat: 276,785,000 bytes of Mathematica numerics.
• 2211.01987, anc/LaminatedK12-face-catalog.tar.xz: 169,076,884 bytes; 7z certificates in 2507.01596, 2602.17412 and 2010.11664 contain 49,486,732, 46,078,740 and 26,709,590 bytes respectively.
• 2607.11953: 6,725 files under anc/hidden-automata-rl/, the largest ancillary file count for one paper, with a reproduction README.

Fluid-dynamics papers account for 23,160,227,356 bytes (27.68%) across 473 papers, including 22,329,262,882 video bytes. By paper count the leaders are hep-ph (1,026), hep-th (1,001), cond-mat.mtrl-sci (596), quant-ph (585) and cond-mat.mes-hall (553).

The 149,573 files contain 137,167 distinct blobs and 388,123,271 duplicate raw bytes (0.46%). Nine blobs also occur in ps: seven animated GIFs from 1705.01852, one JPG and one PNG. None occurs in pdf. The 2,193 TeX-related files also appear in latex; most ancillary content is exclusive to source.

Fetching ancillary content

Inventory queries read only the small index columns:

SELECT paper_id, path, size, sha256
FROM 'hf://datasets/secemp9/arxiv-complete/files/*.parquet'
WHERE kind = 'source' AND starts_with(path, 'anc/');
-- 149,573 rows; 83,668,861,836 raw bytes indexed

Selecting whole shards saves little: 1,527 of 1,586 source shards hold an ancillary row. Footer statistics also help little because paths are ordered within each paper rather than globally: 232,644 of 243,083 groups have path ranges overlapping anc/, allowing only 1.13% of compressed content bytes to be excluded by that test.

The actual matching files occupy 11,681 groups, whose content chunks total 387,407,921,335 compressed bytes (5.95% of the content column). DuckDB can read path first and avoid content chunks in groups with no matching row. In a cold-cache test across eight shards spanning the corpus (00037, 00199, 00419, 00700, 00905, 01053, 01200, 01428), it read 1.71 of 31.48 GB (5.43%), versus all 31.48 GB without the filter. Per-shard shares ranged from 1.9% to 12.6%. On shard 00419, filtering on the affected paper IDs instead read 1.33 GB versus 0.29 GB for the path predicate.

SELECT paper_id, path, size, content
FROM 'hf://datasets/secemp9/arxiv-complete/source/*.parquet'
WHERE starts_with(path, 'anc/');

Budget roughly 390 GB of Parquet reads for this approach. That delivers 83.67 GB of raw ancillary content along with the cost of neighbouring rows in the same groups; the ratio is not a compression ratio. A Python filter inside a datasets stream reads content before applying the filter and therefore traverses the entire 6.51 TB. This distinction also matters for deduplication.

LaTeX and duplication

LaTeX composition

latex is exactly the subset of source with one of thirteen TeX-related extensions. Both directions of the (paper_id, path) set comparison match, as do the bytes and digests. This lets readers obtain TeX-related files in 160.23 GB of Parquet instead of downloading all 6.51 TB of source.

Bibliography databases account for more raw bytes than document source:

Distinct blobs and papers cannot be summed by extension: 1,828 blobs appear under multiple extensions, so the per-extension blob counts total 9,269,598 rather than 9,267,670. Paper counts likewise overlap.

The cls, sty, bst, clo, cfg, ins, dtx and def files total 133,331,761,138 raw bytes, 92.3% of which repeat other files in the config. The most repeated blob is IEEEtran.cls V1.8b: 281,957 bytes in 60,967 copies across 58,901 papers. Within the whole config, 75.06% of rows are distinct blobs and 451,848,934,278 bytes (55.37%) are duplicates.

Bundled bibliography databases

Files selected by ext = 'bib' AND contains(lower(path), 'anthology') account for 14,162 rows, 348,831,785,014 raw bytes, 2,816 distinct blobs and 13,597 papers. They contain 81.2% of .bib bytes and 42.74% of all latex bytes; one copy of each blob would total 34,906,441,967 bytes. These are revisions and variants of the ACL Anthology bibliography bundled with submissions.

The substring definition includes acl_anthology.bib, aclanthology.bib, filtered_anthology.bib and ACL_Anthology_part_aa.bib. Restricting instead to basenames beginning anthology and ending .bib gives 14,087 rows, 346,873,332,048 bytes, 2,781 blobs and 13,542 papers. The 75-row difference uses 38 spellings, led by acl_anthology.bib (18) and aclanthology.bib (13). No match arises solely from an anthology directory component.

There are 8,633 .bib files over 10 MB, totalling 358,326,271,199 raw bytes, or 83.4% of .bib. They represent only 1,028 blobs; 88.8% of their bytes are duplicate copies. Of these rows, 8,317 match the anthology substring test (8,270 under the stricter prefix test). The other 316 rows contain 9,804,072,995 bytes in 192 blobs, led by crypto.bib at 112 copies and 3.57 GB.

One 43,619,593-byte anthology.bib occurs 2,095 times across 2,077 papers and 42 path spellings within latex: 2,059 papers have one copy and 18 have two. It contributes 91,339,427,742 duplicate bytes, 11.2% of the config's raw bytes and 20.2% of its duplication. In source, the same blob has 2,099 copies across 2,081 papers and 46 spellings; the four additional copies have .bak, .txt or extensionless names.

The twenty most copied .bib blobs are all anthology variants, together contributing 230,829,650,496 duplicate bytes (51.1% of latex duplication). All 1,028 over-10-MB blobs contribute 318,277,378,364 duplicate bytes: 98.5% of .bib duplication and 70.4% of the config's. The largest file is the 72,454,841-byte anthology bibliography in 2508.04390.

The large bibliographies occur in 8,441 papers (0.295% of latex papers). Those papers' files total 360,539,676,092 raw bytes (44.2% of the config). Removing just the 8,633 large .bib rows leaves 12,339,449 files and 457,784,199,581 raw bytes.

Filtered copies and download cost

The release audit re-encoded all 80 shards with zstd level 3, 2,000 rows per group and dictionary encoding for ext and paper_id. The table gives the measured sizes of those rewritten copies. Transfer costs for filtering the existing shards are discussed below.

A control re-encoding agreed with the published size within 0.014%; the experimental copies were measured and deleted. Dropping all .bib saves 67,161,051,372 Parquet bytes (41.9%), whereas deduplication saves 65,950,601,244 (41.2%). Those are different filters. The large-bibliography subset's raw-byte saving above does not establish its compressed saving.

The tex-only output is 63.7 GB, but a query cannot obtain it for that transfer cost from the current layout. All 6,370 row groups contain multiple extensions. Only one group's ext statistics exclude tex, allowing 5,899,008 of 159,671,084,826 compressed content bytes to be skipped (0.0037%). Selecting tex therefore still reads nearly all content chunks. If content is unnecessary, the other five columns total only 550,504,989 compressed bytes.

SELECT ext, count(*) AS n_files, sum(size) AS bytes,
       count(DISTINCT sha256) AS blobs, count(DISTINCT paper_id) AS papers
FROM 'hf://datasets/secemp9/arxiv-complete/latex/*.parquet'
GROUP BY ext ORDER BY bytes DESC;

SELECT count(*) AS n_files, sum(size) AS bytes
FROM 'hf://datasets/secemp9/arxiv-complete/latex/*.parquet'
WHERE NOT (ext = 'bib' AND size > 10000000);
-- 12,339,449 files; 457,784,199,581 raw bytes

Duplication by config

Every file occurrence is emitted, even when its bytes duplicate another file. These figures group by sha256 within each config and sum raw content bytes before Parquet compression:

In pdf, 4,859 blobs repeat, none more than five times; 4,799 repeat only within one paper. Byte duplication differs from superseded revisions: 36.6% of PDF raw bytes are older held revisions, while only 0.088% are duplicate bytes. Use version flags for the former and digests for the latter. Repeated paper_text strings have their own analysis.

latex adds no files to source: its 451.85 GB of duplication is already included in source's 811.06 GB. Downloading both configs adds 160.23 GB of Parquet transfer to receive 816.11 GB of raw content again.

The html family is excluded because it has no content config; its inventory contains 11,927 rows, 9,786 distinct blobs and 233,142,152 raw bytes. paper_text contains assembled strings and is outside these file-level totals.

The per-kind blob counts sum to 48,473,838, exceeding the corpus total by 460,209 shared blobs: 252,421 occur in both source and pdf, 207,788 in both source and ps, and none in both pdf and ps.

The source/pdf overlap contains 428,293,962,977 distinct raw bytes. This differs from the 428,146,685,889 bytes of PDF-named source rows in packages without TeX: they are different populations and aggregation units, despite differing by only 147,277,088 bytes. They should not be substituted or added.

On the source side of the PDF overlap, 252,468 of 252,495 occurrences are PDF-named; the others are 25 .bin, one .txt and one extensionless file. They span 252,291 papers, and 252,061 of 252,548 (blob, paper) pairings match the same paper across both configs. The PostScript overlap largely contains submission figures, with source-side occurrences named .eps 116,034 times, .ps 102,148, .png 70,429, .jpg 21,275, .jpeg 13,230 and .gif 12,241.

Similar overlap prevents adding extension-level dedup totals. The LaTeX subset and its complement in source contain 451,848,934,278 and 358,786,357,231 duplicate bytes when deduplicated separately. Their sum is 421,565,652 below source's total because 4,953 blobs get a first copy in both subsets. Deduplicate at the scope you intend to report.

Frequently repeated files

The 311-byte file points an unzipped PowerPoint slide layout to its master. All 89,664 paths contain _rels/ and slideLayout, under 20,486 distinct spellings. Its copies occupy 27,885,504 bytes, of which 27,885,193 are duplicates. 00README.json records the top-level TeX file, TeX Live year and compiler.

By duplicate bytes, the leading source blob is the 43,619,593-byte anthology bibliography: 2,099 copies contribute 91,513,906,114 duplicate bytes (11.28% of source duplication). Other revisions take the next three places. Among source's 1,819,513 repeated blobs, the top 10 account for 28.48% of duplicate bytes, the top 100 for 41.17% and the top 1,000 for 57.75%. IEEEtran.cls contributes 17,204,170,269 duplicate bytes. The first non-LaTeX entry is IEEEtran_HOWTO.pdf: 671,626 bytes in 8,120 copies, or 5,452,931,494 duplicate bytes.

Scope explains the 61,018 corpus-wide versus 60,967 latex copies of IEEEtran.cls: 51 copies across 50 papers have other names, including .bak, .old, .txt, misspellings and eight extensionless IEEEtran files. Within ps, the most repeated blob is the 286-byte example.eps, a handwritten grey-square example dated 1997, with 2,488 copies across 2,035 papers. These representative blobs were read back and re-hashed.

Deduplication and transfer

Plan deduplication against the 2.4 GB files index. This query also selects a real representative (paper_id, path) pair for each digest:

SELECT sha256, any_value(size) AS bytes, count(*) AS copies,
       min(paper_id) AS a_paper, arg_min(path, paper_id) AS a_path
FROM 'hf://datasets/secemp9/arxiv-complete/files/*.parquet'
WHERE kind = 'source'
GROUP BY sha256;
-- 41,302,729 blobs; 8,707,420,317,499 raw bytes

Independent min(paper_id) and min(path) expressions can select values from different rows. arg_min keeps the path associated with the selected minimum paper ID. A streaming pass can instead retain the first occurrence:

from datasets import load_dataset

seen = set()
ds = load_dataset(
    "secemp9/arxiv-complete", "source", split="train", streaming=True
)
# A digest is added once, yielding 41,302,729 distinct blobs on a full pass.
for row in ds:
    if row["sha256"] in seen:
        continue
    seen.add(row["sha256"])
    print(row["paper_id"], row["path"], row["sha256"])

This skips 6,070,875 output rows and 811,056,857,161 raw bytes after transfer. Run to exhaustion or use a reader unaffected by the early-exit issue.

The full set of 41,302,729 SHA-256 strings peaked at 7,425 MiB process RSS in the audit. Truncating to 128-bit strings measured 6,169 MiB and to 64-bit integers 4,935 MiB, but truncated hashes weaken collision protection. The example retains the full digest.

Stored sha256 is a separate column: readers can evaluate it without reading content. The limit here is layout. Under the first-occurrence strategy, almost every row group contains a blob needed somewhere in the output:

The most duplicate-heavy latex group is 98.3% duplicate but still contains new content; no PDF group exceeds 25%. Whole-shard selection cannot exclude any of the 3,686 shards under this strategy. This census does not prove an optimal fetch plan for every possible choice of representative copies.

A Python seen filter saves output storage after the rows arrive. The three primary file configs still transfer 15.84 TB of Parquet in a full stream. A column-aware reader can avoid content chunks with no selected row, as the ancillary query demonstrates.

Re-encoded latex retained 58.8% of the published Parquet size after removing 55.4% of raw bytes. The removed bytes had compressed at 6.85:1; retained bytes at 3.86:1. source was not re-encoded, so its compressed deduplication saving was not measured. The measured raw result for source/PDF/PS is 22.57 TB emitted versus 21.29 TB of distinct blobs.

Schema

All configs use the train split. Content columns use Arrow large_binary for file bytes and large_string for assembled text.

In files, kind identifies the artifact family and source its origin. path is relative to the artifact's source layout. size and sha256 describe the published file bytes. stored_as, stored_size and stored_sha256 identify the original gzip-wrapped mirror object where it differs; they are null otherwise. These three columns exist only in files.

version is an integer and is null throughout source; latex has no version column. S3 supplies one unlabelled source payload per paper. Timestamp-based attribution was abandoned after matching only 6.4%. source_version_proven instead records a source file matching a versioned PDF byte for byte: it is true for 252,019 version rows (5.01%), all with has_pdf, across 252,019 distinct papers. No paper has more than one version marked by this method.

Stored values

String filters are case-sensitive. In particular, latex.ext uses tex, not .tex, and the two encoding columns spell UTF-8 differently.

ext is the lowercased final dotted alphanumeric component of path, with no leading dot, nulls, empty values or surrounding whitespace. All 12,348,082 rows agree with this extraction. Counts by extension are given in the LaTeX composition table.

SELECT count(*)
FROM 'hf://datasets/secemp9/arxiv-complete/latex/*.parquet'
WHERE ext = 'tex';
-- 6,997,048; using '.tex' would return a count of 0

Of the 3,148,796 papers, 1,521,194 have multiple categories. Exact equality with cs.LG selects 35,714 papers; testing membership in the space-separated field selects 283,149. For this literal, contains(categories, 'cs.LG') reproduces that count. When generalising to other strings, match complete category tokens rather than arbitrary substrings.

SELECT count(*)
FROM 'hf://datasets/secemp9/arxiv-complete/metadata/*.parquet'
WHERE list_contains(string_split(categories, ' '), 'cs.LG');
-- 283,149

OAI set specs use a different grammar: cs.LG maps to cs:cs:LG, hep-ph to physics:hep-ph and cond-mat.mtrl-sci to physics:cond-mat:mtrl-sci. Of the 158 specs, 126 repeat the archive name, 20 nest a physics sub-archive and 12 have two components. Read the stored list rather than constructing a spec from a category string.

Some source paths were not valid UTF-8, including CP949 directory names. path_encoding = 'percent' marks their percent-encoded representation; urllib.parse.unquote_to_bytes recovers the original path bytes. For literal substring matching, contains() avoids the _ and % wildcards in SQL LIKE.

The nine recorded licence URLs, with metadata-paper counts, are:

CC BY 4.0 is not a stored value. The same paper-level value is copied into versions, paper_text and sample; see licence populations for the different denominators and LICENSE for reuse terms.

Field completeness

Counts below cover all metadata rows. Populated means non-null and non-empty; it does not exclude sentinel strings such as None.

oai_sets is a list and is never empty. submitter IS NOT NULL retains all rows, including 15,090 empty strings; submitter <> '' excludes them. Of those empty submitters, 15,087 predate 2004. Live records for acc-phys/9411002 and 2001.10551 on 2026-09-14 also contained empty submitter elements.

DOI and journal reference

DOI coverage varies by field and age. Filtering on doi IS NOT NULL selects 42.23% of the corpus and disproportionately retains older physics papers. Rows below group primary categories by prefix and date by first submission:

Other overall rates include hep-th 73.90%, gr-qc 70.77%, hep-ph 69.61%, quant-ph 61.23%, stat 16.74% and eess 19.78%. The recent decline is consistent with publication and metadata-update lag; these percentages do not by themselves identify its cause or measure eventual publication rates.

doi and journal_ref overlap but neither contains the other: 849,421 papers have both, 480,298 only a DOI, 112,841 only a journal reference and 1,706,236 neither. Their union is 1,442,560 papers (45.81%) with recorded publication information. A paper's absence from this set does not establish that it is unpublished.

Classification codes and sentinel values

acm_class has 63,309 non-null values, but 38,203 are the literal string None and 112 are empty. The remaining 24,994 (0.79% of papers) are all byte-identical to the corresponding msc_class. The combined fields add no papers beyond the 397,244 with a code after excluding None and empty values.

Live arXivRaw checks on 2026-09-14 reproduced this duplication for cs/0205064, math/0701684 and 2608.23263; q-bio/0312017 returned <acm-class>None</acm-class> with no MSC element. Both columns preserve the upstream values. Column names therefore do not reliably identify the coding scheme: cs/0205064 has ACM codes F.2.2, F.1.1 in both, and math/0701684 has MSC codes in both. Of the 24,994, 24,190 begin with a digit and 407 with a capital letter followed by a dot.

SELECT paper_id, msc_class, acm_class
FROM 'hf://datasets/secemp9/arxiv-complete/metadata/*.parquet'
WHERE acm_class NOT IN ('None', '');
-- 24,994 rows

After lowercasing and trimming surrounding whitespace, the sentinel values none, na, n/a and - also occur in comments (191), msc_class (89), report_no (15) and journal_ref (1). Adding no, nil, null, -- and x gives 237, 101, 16 and 1. These are too few to change the printed completeness percentages, but applications should handle them.

Classification fields are also discipline-specific. Of the 397,244 code-bearing papers, 340,962 (85.8%) have math primary categories; 53.92% of 632,299 mathematics papers have a code. Computer science accounts for 14,608 of the 24,994 non-sentinel acm_class rows (58.45%). Absence of a classification code should not be treated as a uniform quality defect.

Proxy and version fields

proxy records a submission agent acting on an author's behalf. Its 57,192 rows contain 10,659 distinct values. The top five account for 37,823 (66.1%): ccsd 25,387, vtex 4,963, EPTCS 4,247, auai 1,899 and Sigma 1,327. Spelling varies (Sigma, sigma, Journal Sigma), so normalisation may be needed. A live check for quant-ph/0402145 on 2026-09-14 reproduced ccsd.

Only two of versions' nineteen columns contain nulls:

pdf_sha256 is null exactly when has_pdf is false. None of the eight boolean columns is null, and no string column contains an empty string.

Metadata completeness by first-submission date

first posted	papers	licence	comments	DOI	journal ref	MSC	report no	ACM	proxy
1991–1999	120,745	0.24%	96.80%	65.39%	54.34%	3.24%	38.47%	0.41%	0.01%
2000–2007	335,945	2.53%	92.03%	63.71%	55.18%	12.86%	14.25%	1.28%	2.35%
2008–2014	544,610	99.11%	81.43%	57.93%	41.92%	16.40%	7.48%	1.55%	4.24%
2015–2019	638,645	100%	71.29%	49.01%	33.17%	14.32%	3.89%	1.40%	1.82%
2020–2023	755,085	100%	66.04%	37.48%	23.29%	11.95%	2.61%	2.38%	1.13%
2024–2026	753,766	100%	58.94%	16.61%	12.64%	10.49%	1.76%	3.05%	0.80%
all	3,148,796	85.62%	72.02%	42.23%	30.56%	12.62%	6.12%	2.01%	1.82%

As in the main table, non-empty sentinel strings count as populated. After excluding None, ACM rates become 0.00%, 0.00%, 0.53%, 0.63%, 1.09% and 1.31% respectively, for 24,994 papers overall.

The licence boundary is particularly sharp: all 2,682,895 papers first posted on or after 2008-03-01 have a recorded licence. Of the 465,901 earlier papers, 452,715 (97.2%) do not; the latest first-submission date among papers with a null licence is 2008-02-29. A licence-based selection therefore changes the corpus's age distribution substantially.

SELECT count(*) AS n_rows,
       count(doi) AS doi_not_null,
       count(submitter) AS sub_not_null,
       count(*) FILTER (WHERE submitter <> '') AS sub_not_empty,
       count(acm_class) AS acm_not_null,
       count(*) FILTER (WHERE acm_class NOT IN ('None','')) AS acm_real
FROM 'hf://datasets/secemp9/arxiv-complete/metadata/*.parquet';
-- 3,148,796; 1,329,719; 3,148,796; 3,133,706; 63,309; 24,994

Provenance and verification

Content handling

No missing PDF was generated by compiling TeX or converting another representation. File contents were preserved, with three PDF replacements and removal of gzip transport wrappers as described below. paper_text is a derived, decoded string; its assembly is documented separately.

Three corrupt mirrored PDFs were replaced with complete copies from arXiv:

The redirect pointed to browse.arxiv.org/pdf/2401.18030v3. Reconstructing its recorded 267 bytes reproduces both the repair-journal SHA-256 prefix 2e066757… and the mirror-manifest MD5 prefix db732060…, establishing what the replaced object contained. The published replacement is 321,907 bytes with a PDF header and end marker.

No published column marks these three replacements. Their source remains gcs, and their paths retain the mirror layout. Published size, sha256 and versions.pdf_sha256 describe the replacement bytes. repaired_from existed only in an internal index; select these three explicit (paper_id, version) pairs to identify them. The 6,475 gap-filling PDFs instead carry source = 'web_scrape'.

Gzip wrappers were removed from 2,168,515 objects, all kind = 'ps': 2,070,188 named .ps.gz, 90,409 .eps.gz and 7,918 with other names. The original name, size and digest remain in files.stored_as, stored_size and stored_sha256.

Of those objects, 2,168,162 grew when decompressed, one stayed the same length and 352 shrank. The shrinking objects are 3–154 bytes, across 27 papers: 41,149 bytes published versus 46,101 stored, with a maximum reduction of 39 bytes. Gzip overhead can exceed its compression saving for small inputs. All 62 objects published at 64 bytes or fewer shrank; 290 of 786 above 64 and below 256 bytes did; none of the 2,167,667 at 256 bytes or more did. The equal-size object is 2201.10203 v1's 179-byte gyro_Yao_PRL_Revised.out.ps. Consequently, stored_size need not be less than size.

The build stored identical blobs once internally, but published configs emit every occurrence. No submission file was removed solely because another paper contained the same bytes.

The 54 flagged PDF candidates

The build inspected the first kilobyte and last two kilobytes of all 19,227,355 objects named .pdf case-insensitively. It flagged 13,968: 13,035 without a header, 752 without an end marker in the tail, 179 with a header at a non-zero offset and two empty files. Of these, 13,871 were inside source packages. The remaining 97 rows represented 95 distinct directly held objects; two hardlinked files were encountered twice.

Of the 95 objects, 28 had offset headers and 13 were PostScript figures named .pdf. The other 54 candidates entered a comparison against arxiv.org on 2026-09-05, at fifteen-second request intervals. Not all 54 were defective PDFs:

The 42 retained PDFs span 42 papers and 58,461,737 bytes. Each had a %PDF- header but no %%EOF in its last two kilobytes, and each live comparison returned an identical SHA-256. This establishes that arxiv.org served the same bytes on that date; it does not establish that no other copy exists. No schema column flags these files.

Two of them, 1304.1318 v1 and 1311.4268 v1, are web_scrape rows: arxiv.org served those bytes on August 30 and again on September 5. The observed defects therefore predate dataset packing.

The eight misnamed figures are five distinct blobs across four papers, totalling 1,091,027 bytes:

The comparison script failed to extract an identifier from their ps/ paths, then requested /pdf/None eight times. Those 404s say nothing about the figures' availability. All four papers have their rendered PDFs for every version reported in the snapshot. These figures are eight of the 21 PDF-named objects in ps, none of which is actually PDF.

The temporary-file candidate was tmp.1702.02106v3.pdf, 18,190,224 bytes without an end marker. Its path is absent from files. The published 1702.02106v3.pdf is the complete 20,092,270-byte mirror object (sha256 8b41aea2…), with thirteen readable pages and a valid 126-entry cross-reference table at offset 20,089,591. Its row was read directly from pdf/train-00205-of-01886.parquet and matched the recorded digest.

arxiv.org returned a different complete 20,091,658-byte rendering (sha256 5207ce79…) on September 5 and again September 14. Thus the comparison's fourth “repairable” classification was a temporary-file false positive. Of the 54 candidates, 45 were defective PDF objects: 42 retained and three replaced. The other nine were not defective published PDFs.

Verification

The build audit read, decompressed and re-hashed 69,789,708 content rows across 3,736 shards, finding zero digest mismatches and zero unreadable shards. Digests had been recorded during indexing, before packing.

The separate 991-row sample shard is outside that total. All 991 rows match paper_text on paper ID and text digest, and their published strings also pass direct UTF-8 digest verification. The three index configs contain no file-content column.

The byte totals refer to different scopes:

The last two round to 23.39 TB and 23.63 TB. Neither is the 16.08 TB compressed download total.

For the four file-level content configs, structural checks reconciled row counts, (paper_id, path) sets in both directions and every stored byte length against the corresponding inventory. No missing or invented rows were found. Shards were present, readable, sorted and non-overlapping, with no duplicate keys.

For a file row, hash content. For an assembled text row, hash its UTF-8 encoding:

import hashlib

def verify_file_row(row):
    return (
        len(row["content"]) == row["size"]
        and hashlib.sha256(row["content"]).hexdigest() == row["sha256"]
    )

def verify_text_row(row):
    return (
        hashlib.sha256(row["text"].encode("utf-8")).hexdigest()
        == row["text_sha256"]
    )

text_sha256 validates the published string. Source files have separate digests in latex. For the 191,366 latin-1 fallback rows, re-encoding as UTF-8 changes the non-ASCII bytes from those presented to the decoder. Use the individual latex rows and their digests to verify original source bytes.

Verifying your download

SHA256SUMS at the repository root lists every Parquet file. From a complete download's root directory:

sha256sum -c SHA256SUMS

For a filtered download, with the manifest available in the same root:

sha256sum --ignore-missing -c SHA256SUMS

The latter checks only files that are present; it does not certify that the requested subset is complete. File checksums verify transfer integrity, while row-level digests verify the content inside Parquet.

File quality and omissions

The sections below distinguish absent files, incomplete contents and unusual but valid files. None of these conditions has a general-purpose quality flag in the schema. The coverage section describes fileless papers and version gaps; text quality describes short rows and resolver limitations.

PDF-only source

There are 265,111 source-bearing papers without .tex or .ltx, so they do not appear in paper_text. Of these, 261,052 are also absent from latex; the remaining 4,059 have other TeX-related files, including .cls in 3,039 papers, .sty in 1,669 and .bbl in 908. These counts overlap. Testing only for .tex gives 265,610 papers because 499 use .ltx instead.

Most of this group submits the paper as PDF: 251,258 (94.8%) have exactly one source file and it is a PDF. In total, 252,815 have a PDF in source, 259,382 have PDF or PostScript, and 264,717 have an arXiv-rendered PDF in pdf. PDF-named source files contribute 428,146,685,889 of the group's 439,623,556,691 raw bytes. These figures do not count images alone; source PDFs can contain the full paper.

The group also includes other cases. In particular, 4,090 papers have .cry files described below, and extensionless documents account for part of the TeX-name shortfall. Absence of recognised extensions does not prove absence of TeX content.

metadata minus latex is a different set of 288,510 papers: it also includes 13,898 papers without any files and 13,560 with no source package.

.cry files

source contains 7,387 .cry files across 4,393 papers, totalling 358,179,037 raw bytes and 7,385 distinct blobs. Most names append .cry to a TeX-related suffix: 6,675 .tex.cry, 187 .pstex_t.cry, 142 .bbl.cry, 119 .sty.cry, 69 .cls.cry, 26 .bst.cry and a longer tail. Four have no preceding dotted component. None enters latex through its .cry suffix.

A systematic sample of 247 files, one in thirty in (paper_id, path) order, was read from published Parquet and re-hashed with zero size or digest mismatches. Median entropy was 7.995 bits per byte (range 5.857–7.999), with 52–71% non-printable bytes. No sampled file had a recognised signature or decompressed as gzip, zlib, bzip2 or xz.

Several comparisons support an encrypted TeX interpretation for these TeX-named files:

• Three files have a same-named cleartext twin in the same package, two in 1007.0328 and one in 0902.4344. Each pair has equal lengths and different digests.
• All 22 boxedeps.tex.cry files are 33,533 bytes, matching a clear boxedeps.tex blob, but all 22 encrypted blobs differ. Sixteen ieeetran.cls.cry files have the 201,353-byte length shared by 7,937 clear IEEEtran.cls files.
• The 122 includepdf.tex.cry files have only 24 distinct 97-byte prefixes. Prefix groups correspond to neighbouring IDs posted within minutes, such as 1205.6691–1205.6699 over 82 seconds and 1203.6400–1203.6406 in under three seconds. Whole blobs also repeat in neighbouring papers: 1005.1062/1005.1065 share an IEEEtran.cls.cry, and 1211.1505/1211.1506 share macros.tex.cry.

For 1203.6400–1203.6406, each package contains a PDF and a 143–150-byte encrypted wrapper. Reconstructing the wrapper template with each package's PDF filename gives the matching length. XOR comparisons recover the shared prefix stream and the other filenames, supporting the interpretation of transformed LaTeX. The streams diverge after differing input: 1203.6404 and 1203.6405 name p646_goetzgraefe_vldb2012.pdf and p656_goetzgraefe_vldb2012.pdf, differing at offset 99; their encrypted files agree through the first 99 bytes and differ thereafter.

The key source, transform and reason for these files' presence in arXiv's archives were not established. No published key or general decoder is available. The known-plaintext experiment does not recover arbitrary files.

Of the 4,393 papers, 4,090 lack plain .tex and .ltx; 4,088 of those contain .tex.cry. All 4,393 have a rendered PDF in pdf, so this affects source access rather than PDF availability. First versions run from 1995 to January 2013, apart from 2602.15793, whose ancillary .cry file may be an unrelated use of the suffix.

PostScript content

ps means objects held under the mirror's /ps/ tree. It contains both rendered PostScript and submission assets. After removing the <paper_id>v<n>. prefix from each basename, 1,944,222 of 2,237,839 rows (86.9%) are the version's own <paper_id>v<n>.ps render. The other 293,617 (13.1%) are assets across 60,609 papers, totalling 61.57 GB, or 2.30% of the config's 2.68 TB raw content.

The fourteen PostScript spellings total 218,427, with another 119 numbered variants bringing that group to 218,546. The sixteen other spellings plus extensionless files total 71,580, with another 3,491 objects across 1,214 spellings bringing that group to 75,071. Together the groups give 293,617. Many tail suffixes are bare numbers from figure names.

Of the 1,953,094 indexed PostScript version pairs, 8,872 have assets but no render. They contain 29,023 assets across 7,397 papers. This explains why the 293,617 assets exceed the 284,745 rows above one per pair: 293,617 − 284,745 = 8,872. Subtracting one row per pair otherwise assumes a render where none exists.

Byte checks covered 1,431 objects. All 400 sampled renders and 385 sampled PostScript-named assets began %!PS. So did 452 of 646 assets sampled from other names, including every sampled .tex, .prn, .xcp, .pfb, .bb, .enc and .pro; small .ai, .tikz, .plt, .md, .bib, .old, .ep and .post groups were read in full. All 21 files named .pdf were also read: thirteen were PostScript, five PNG and three JPEG.

Raster-named assets total 69,195 files and 4.57 GB, 0.17% of raw config bytes. Of 182 sampled, 179 matched the extension and three were PNGs named .jpg or .jpeg. Extension counts are exhaustive; sampled signatures do not establish the actual format of every untested file.

PostScript contributes little additional version coverage beyond PDF: 159 version pairs across 137 papers have ps but no PDF at that version. Of those papers, 63 have a PDF at another version and 74 have none at any version. Thus the approximately 99.99% overlap concerns version coverage. The pdf and ps configs share zero byte-identical blobs.

Filename and format mismatches

Extensions are useful selection fields but do not certify format. The build's PDF-name census examined 19,227,355 objects and found 13,035 without a %PDF- header in the first kilobyte. Of these, 13,013 are published source files: 0.091% of the 14,246,890 PDF-named source rows, across 3,116 papers, totalling 3,533,363,287 raw bytes. Every one matched a published row and size.

Of the 184 NUL-prefixed files, 109 are entirely NUL-filled: 16,509,598 bytes across 23 papers. They are a subset of the corpus-wide 913 NUL-filled files.

A published-Parquet recheck read 3,966 flagged source files from 400 shards and 1,435 row groups, with zero length or digest mismatches. This was 28.6% of the 13,871 source files flagged by all PDF checks (13,013 without a header, 705 without a tail marker, 151 with offset headers and two empty). The 13,013 format mismatches are clustered: 53% occur in papers holding at least ten, and 2009.08576 alone contributes 334.

The other 22 no-header objects are the 21 misnamed figures in ps and the 267-byte redirect for 2401.18030 v3, which was replaced. The published PDF tree contains one non-PDF, 1611.09018v3.txt; all its .pdf-named objects have PDF headers, though some remain defective.

A separate uniform sample of 900 source row groups covered 176,417 rows. Of these, 164,663 used the 24 figure and TeX extensions below:

The sample contained 339 mismatches (0.21%) among the selected extensions. Other formats were outside its scope. Its PDF mismatch rate (0.033%) differs from the full census (0.091%); use the census for that population. The JPEG estimate rests on only seventeen mismatches.

A minimal signature helper for bytes already in memory is:

MAGIC = (
    (b"\x89PNG\r\n\x1a\n", "png"), (b"\xff\xd8\xff", "jpeg"),
    (b"GIF87a", "gif"), (b"GIF89a", "gif"),
    (b"%!PS", "ps"), (b"\xc5\xd0\xd3\xc6", "ps"),
    (b"II*\x00", "tiff"), (b"MM\x00*", "tiff"),
    (b"BM", "bmp"), (b"\x1f\x8b", "gzip"),
    (b"PK\x03\x04", "zip"), (b"Rar!", "rar"),
    (b"\x00\x05\x16\x07", "appledouble"),
)

def sniff(content):
    if b"%PDF-" in content[:1024]:
        return "pdf"
    # Each listed prefix identifies a candidate format, not file validity.
    for signature, kind in MAGIC:
        if content.startswith(signature):
            return kind
    return None

This is not a validator or a complete format detector. It allows PDF headers at a non-zero offset, unlike an offset-zero test. A bare %! is ambiguous: %!TEX root = ... is a TeX editor directive, while %! alone can head PostScript. EPS can also begin with a bounding box or operators. P7 identifies xv thumbnails. A previous detector treating every %! as PostScript inflated the sampled TeX mismatch rate to 1.17%; the corrected rate is 0.04%. Case-fold filename suffixes: 4,257 of the 19,220,877 PDF-named paths in the published inventory use mixed or uppercase suffixes.

PDF end markers

The original scan flagged 752 rows with a header but no %%EOF in the last two kilobytes: 750 distinct objects, because two hardlinked objects were encountered twice. Of the flagged rows, 705 were inside source packages; 47 described directly held objects, of which 45 were distinct.

A recheck of that flag list classified 495 rows as truncated, with no %%EOF anywhere. In the other 257, the marker was followed by data: 181 had padding, 49 binary data and 27 an appended HTML page. Many PDF readers tolerate trailing data, so a missing tail marker alone is not proof of truncation. Another 179 rows had an offset header (151 in source); an offset-zero signature test would misclassify them.

Two PDF-named source rows were empty: 2309.07108's stdin.pdf and 2505.19488's images/transformer.pdf.

Of the 752 flagged rows, 58 had exact power-of-two lengths: seventeen at 1 MiB, fifteen at 64 KiB, fourteen at 512 KiB, four at 256 KiB, three each at 128 KiB and 32 KiB, one at 2 MiB and one at sixteen bytes (2403.07721, figures/fig-overview.pdf). This pattern is consistent with buffer-boundary truncation; it does not identify the responsible component. Fifteen were rendered PDFs returned byte-identically by arxiv.org during rechecks, 42 were source-package files and one was a repeated encounter of a rendered object. The direct-object outcomes are accounted for in the 54-candidate analysis.

PDF rendering changes

There are 1,342 pdf objects over 64,000,000 bytes, across 1,071 papers: 0.027% of rows but 150.7 GB, or 1.45% of PDF raw content. Of these, 250 exceed 128 MB, 69 exceed 256 MB and sixteen exceed 512 MB. The largest, 2206.04656v2, is 1,113 MB versus a config median of 0.64 MB.

On 2026-09-13, the audit compared sizes using HEAD requests to the exact version URLs for the eight largest objects and 32 randomly selected from the remaining 1,334. All eight largest were bigger in the snapshot, by 9.1–66.6 times. Among the random 32, 26 had matching reported sizes, one differed by 218 bytes and five were 8.5–21.8 times larger here. A matching size does not prove byte identity. The other 1,302 were not tested.

For the pair examined in detail, 2206.04656v2, both files have the same 28 pages and 85 images at identical pixel dimensions. The live file was 21.7 MB, with a render date of 2023-02-11 versus 2022-06-13 here, and compressed image streams stored essentially uncompressed in the snapshot. That finding applies to this pair. arxiv_pdf_url provides the live route when a current rendering is preferable to exact snapshot bytes.

HTML omission

files indexes 11,927 html objects across 1,459 papers and 233,142,152 raw bytes, all from gcs, but there is no html content config. The family was carried through indexing and omitted from the packer's config list; the build records no deliberate exclusion decision. All 11,927 held objects were re-read and matched their recorded sizes and digests during the audit. They could be published in a later revision.

The family comprises pages and assets in 2,024 version directories. A full signature census found 9,388 raster images (8,544 GIF, 601 JPEG, 242 PNG, one WebP), 106 PostScript files, eleven PDFs, 2,407 markup files (2,364 documents and 43 fragments), four PCL streams and eleven other text/source files. There are 2,406 HTML-named paths.

The 11,927 rows contain 9,786 distinct blobs and 27,287,667 duplicate bytes (11.70%). For example, quant-ph/9909090 v1 stores one 60-byte GIF under sixteen names. Some bytes are available through source copies: the four PCL streams in patt-sol/9303002, patt-sol/9303004, patt-sol/9303005 and patt-sol/9304001 match files in source, totalling 7,525,635 bytes. Source contains eight .pcl files across six papers, 7,890,617 bytes in total.

Of the 1,459 HTML-bearing papers, 1,024 have paper_text, 109 have a PDF, 45 have PostScript, 382 ship HTML in source (518 files, 18,092,706 bytes), 52 ship a PDF in source and 38 ship PostScript in source. These sets overlap. Only 2608.02072, the LOCO 2026 workshop proceedings, has no published text route: its sole held object is the 6,400-byte arxiv/arxiv/html/2608/2608.02072v1/loco2026-proceedings.html.

has_html is true for 1,988 reported versions: 1,987 without PDF or PostScript across 1,458 papers, plus 0802.0522 v1, which also has PDF. sum(n_html_files) is 11,891. The remaining 36 inventory rows (740,031 bytes) belong to 36 version pairs absent from the OAI record and therefore from versions. These flags describe the held inventory. The HTML content config is absent.

Withdrawal and removal records

Withdrawals and administrative removals appear in five recorded forms: a bare marker, a marker followed by a note, a note under another filename, a PDF notice, and an empty source payload. Filenames affect whether the record reaches paper_text; the note's text identifies who removed the submission and why. Because source payloads are unversioned, the held source may be a replacement notice even when an earlier PDF remains available.

Bare markers

Hash comparisons found 11,497 files containing only %auto-ignore and optional trailing whitespace. Of these, 11,416 contain exactly the twelve bytes, across 11,415 papers, and 81 add whitespace. There are 11,479 .tex-named files and 11,423 named exactly <paper_id>.tex.

In paper_text, 11,430 rows contain this stub: 11,369 are exactly twelve characters and 61 add whitespace. One further row adds a LaTeX \\, giving 11,431 rows at twenty characters or fewer. No row is shorter than twelve characters or empty, blank or null.

For all 11,369 exact-stub papers, the held source package consists solely of the twelve-byte <paper_id>.tex. Each has multiple versions and at least one metadata-only version; for 10,884, that is the newest reported version. metadata.comments contains withdraw for 5,215 of the 11,369 (45.87%), compared with 8,713 of all 3,148,796 papers (0.277%) and 3,498 of the 3,137,427 non-stub papers (0.111%).

For the other 485, the newest version is not metadata-only; 477 have a PDF, consistent with reposting after withdrawal and a stale held source stub. Earlier PDFs remain available for 11,355 of the exact-stub papers: 16,703 rows and 23,533,143,198 raw bytes.

Marker followed by text

There are 13,302 rows beginning with %auto-ignore. Subtracting the 11,430 bare/whitespace stubs leaves 1,872 rows with further text. Of these, 1,317 are under 200 characters and another 362 under 1,000. Examples include author withdrawals, publication problems and administrative removals, such as cs/0003063, astro-ph/0104484 and gr-qc/0502058.

The marker does not always mean withdrawal: 154 of the 13,302 rows exceed 10,000 characters and contain ordinary papers. Treat it as a candidate signal rather than an unconditional exclusion rule.

Notes under other names

A filename audit, combined with inspection of single-file source packages at most 8,192 bytes whose file was not named <paper_id>.tex, identified 97 papers whose submission is a removal or withdrawal note. Ninety-six have only the note; 2603.12203 also has arXiv's 298-byte 00README.json. All notes were read from published Parquet and checked against their hashes and lengths; all begin with %auto-ignore, four after leading whitespace.

Only three enter paper_text: 2305.02155, 2308.13987 and math/0603115, whose notes end in .tex. The other 94 do not, although 88 of the 97 papers have a rendered PDF. Additional notes in hep-th/9502028 and gr-qc/9410009 accompany figures and fall outside the small-package selection. Files named removed.tex or deleted.tex in ordinary multi-file packages can be author drafts rather than removal notices.

Eighty-three notes name arXiv administrators or carry an admin signature; fourteen do not. The administrative reasons are:

• 66: submitter lacked the right to agree to the licence;
• 9: copyright infringement;
• 2: author-policy violations;
• one each: external text overlap, an unconsenting co-author, duplicate submission, inaccurate authorship, names added without authorship, and removal following IEEE Xplore publication.

All 79 removed.txt, both removed.tex and the one withdrawn.txt fall in the administrative group. auto-ignore.txt splits one administrative to three others. The remaining names belong to the unsigned/author group. Examples include physics/0401074 (temporary withdrawal at an advisor's request), cond-mat/0310326 (different results) and 0905.4570 (a mistake in the previous version). q-bio/0606043 does not identify who removed it. Fourteen is the count of notes without an administrative signature.

PDF notices and empty payloads

The audit read and extracted text from all 2,290 source packages consisting of one PDF at most 64 KiB. Sixteen yielded at most 400 characters. Eleven contain only an explicit withdrawal notice, including math/0310247, quant-ph/0609176, cond-mat/0504111, physics/0501147 and quant-ph/0611120; five print %auto-ignore on the page. A twelfth, cond-mat/0211524, promises a later complete version.

The other four produced no extracted text: 1009.4523 has one blank page, quant-ph/0504128 declares zero pages, and cs/0010015 and 2401.11525 could not be opened by the tested pypdf reader. None of these sixteen has a paper_text row, but all have a rendered PDF row.

The fifth recorded form is 1810.03044: its whole source payload is the zero-byte 1810.03044.bin, with an administrative copyright-removal note in metadata. It is one of the 242 zero-byte source files.

The first two forms account for 13,302 of 2,856,227 paper_text rows (0.47%); 11,430 (0.40%) are bare or whitespace-only stubs. These figures include the long marker-bearing rows noted above. The other forms require looking beyond paper_text, and previous PDFs often remain available.

NUL-filled files

There are 913 non-empty files consisting entirely of zero bytes, all in source: 215,358,402 raw bytes across 189 papers and 719 distinct blobs. They account for 0.0017% of indexed files, 0.00095% of indexed raw bytes and 0.0060% of papers. Some are valid empty archives or application records; zero-filled does not always mean damaged.

The census compared each recorded digest with the SHA-256 of a zero-filled buffer of the same length. Computing these digests incrementally over the 3,533,564 distinct sizes requires hashing only as many zero bytes as the largest size, 1,568,567,496. For example:

import hashlib

def zero_hashes(sizes):
    block = b"\x00" * (1024 * 1024)
    digest = hashlib.sha256()
    previous = 0
    # Incremental updates must equal sha256 of n zero bytes at each size n.
    for n in sorted(set(sizes)):
        if n < 0:
            raise ValueError("File sizes must be non-negative")
        remaining = n - previous
        # Chunking preserves that digest while bounding the temporary buffer.
        while remaining:
            take = min(remaining, len(block))
            digest.update(block[:take])
            remaining -= take
        previous = n
        yield n, digest.copy().hexdigest()

Match only size > 0 for the NUL-filled population; size zero belongs to the separate empty-file population below. The index census and independent content-config column checks agree: source 913, latex 86, pdf zero, ps zero. Every one of the 913 was then read from published Parquet across 178 shards and 268 row groups. All lengths and digests matched, and every byte was zero. Distinct lengths and distinct blobs both equal 719.

All 913 were also located in the original unmodified source archives for the 189 papers. Every original member was already entirely NUL-filled; none differed or was missing. This rules out zeroing by this dataset's extraction and packing steps.

The 86 files also in latex contain 36,188,700 bytes across 34 papers: 38 tex, 18 bst, 10 sty, eight tikz, seven bib, two bbl and one each of cls, ins, dtx. 1802.09064 contains 29 of them. Neither paper_text nor sample has an entirely NUL-filled string; their digests were checked against zero-filled strings through length 46,000,000.

The audit classified the files as follows:

The empty archives include 54 files of 1,024 bytes and seventeen of 10,240 bytes, corresponding to tar end blocks and padding. GNU tar's default empty archive is 10,240 zero bytes (sha256 84ff9269…). Of the 71, 52 have .tar or .tgz names and nineteen are extensionless. All 101 NUL-filled files whose length is a multiple of 512 opened as zero-member archives with Python tarfile; that parser result alone does not establish that every such file was intended as an archive.

The 42 Krita layerN.defaultpixel records belong to unpacked Krita documents. Zero is the correct default for a transparent layer. All 42 files with this suffix in the corpus are NUL-filled. Among the remaining files, the largest are 2107.14589's anthology.bib (33,496,382 bytes), 1410.1323's arXiv.tar (27,140,749) and 2209.11629's arxiv.tar (15,405,436).

This population does not show the large power-of-two pattern seen in flagged PDFs. Although 139 files have power-of-two lengths, the largest is 32,768 bytes and 54 are the 1,024-byte empty archives. None is 64 KiB, 128 KiB, 256 KiB, 512 KiB or 1 MiB. The observed alignment is mainly tar's 512-byte blocks; the cause of the other zero-filled content was not established.

All 189 papers have a rendered PDF, and 186 have paper_text. The median affected paper has one NUL-filled file among 32 source files. Three papers have only an empty source tar (1607.05755, 1702.06740, 2008.05843), all the same 10,240-byte blob; two are marked withdrawn in comments. Their PDFs are still present.

Entire directories are affected in three papers: 22 (paper, directory) pairs containing multiple files are all NUL-filled. These are 1802.09064's samples/ (25 files, 2,121,825 bytes) and content/Archived/ (eleven), nineteen imgs/… directories in 2406.05561, and 2406.05558's pictures/DACH (five).

NUL-filled files by common filename suffix

suffix	files	raw bytes	papers
png	327	26,718,446	19
pdf	109	16,509,598	23
jpeg	69	29,496,155	1
tar	63	78,300,878	63
no extension	50	394,229	35
defaultpixel	42	162	5
tex	38	322,975	7
eps	32	14,071,809	9
bst	18	932,355	12
txt	15	13,283	3
jpg	13	4,191,068	6
sty	10	102,885	9
svg	9	465,949	2
bib	7	34,374,365	6

There are 62 suffix spellings in total, plus fifty extensionless files. The 68 .tar/.tgz files and 109 PDF-named files discussed elsewhere are subsets of these 913. Affected papers date from 1997-01-22 to 2026-07-30; 141 first appeared in 2018 or later. Leading categories are cs.LG (14), eess.SY (8), then five categories at seven papers each.

Zero-byte and small files

The inventory contains 242 zero-byte files across 33 papers, all in source, with the empty-content digest e3b0c442…. latex includes seventeen of them across seven papers: thirteen tex, two bib, one bbl and one cfg. There are none in pdf, ps or the indexed HTML family. paper_text and sample have no empty or null strings; their minimum lengths are twelve and 521 characters respectively.

These are separate from non-empty NUL-filled files. Two papers contain both types: 2405.14882 and 2505.19488. Testing only size > 0 excludes empty files but retains zero-filled ones.

The paper counts overlap: 2505.19488 has both Photos sidecars and an empty images/transformer.pdf. Its sidecars include thirteen SQLite WAL files, twelve change journals, three Spotlight shards and two locks. Of 669 __init__.py files corpus-wide, 122 are empty and the others reach 104,648 bytes; 2211.10897 contributes 118 of the empty files. An empty package marker is normal. Other zero-byte files include drafts, bibliographies, build artifacts, fourteen ancillary CNC toolpaths, four GLM headers, three Windows :Zone.Identifier sidecars and the two PDF-named source files identified above.

All 242 were checked against original source archives. Of these, 241 were already empty members of submission tars. For 1810.03044, arXiv's outer payload member 1810/1810.03044.gz in arXiv_src_1810_009.tar was itself empty. No original differed or was missing.

1810.03044 is the fifth form in the withdrawal/removal account: an empty whole submission, with no PDF, PostScript or paper_text row. Metadata identifies an administrative copyright removal. The other 32 papers have both rendered PDFs and paper_text; their packages contain 7–3,229 files, with first submissions from 1996-09-13 to 2026-05-26.

The content-config minima were read from Parquet and re-hashed. The HTML minimum was read from the held build object because no HTML config exists.

Rendered <paper_id>v<n>.ps files have a minimum of 9,241 bytes; none is below 4,096. All tiny ps rows are assets. The 62 assets at most 64 bytes total 2,823 bytes across eighteen papers: 46 bounding-box sidecars, six TeX editor-directive files and ten bodyless PostScript headers. All begin %!, illustrating why that prefix alone is not a reliable format test. The five three-byte objects contain %!\n; one is hep-lat/9204001's figure1_too_big.ps. The same blob occurs eight times across six papers corpus-wide, including three source copies.

All fifteen PDF-config files below 4,096 bytes were read in full and have headers, end markers and readable page trees. Fourteen declare one page. quant-ph/0504128 v1 is a 322-byte, zero-page PDF produced by AFPL Ghostscript 7.04, with /Kids [] /Count 0, a four-object cross-reference table and startxref 167. The same bytes occur in source as quant-ph0504128.pdf. Its metadata says “9 pages, latex, no figures”, but its version is reported at 0kb. The next smallest PDF, 1009.4523 v1, is 714 bytes with one blank page. Small size does not necessarily indicate a misnamed or truncated file.

HTTP error bodies instead occur in source: sixty files across forty papers, 695 bytes in total. These are 45 copies of Not Found, twelve of Internal Server Error, two of 404: Not Found and one Not Found with a newline, found by digest. Their names include fifteen .pdf, thirteen .png, nine .eps, eight .bbl, five .jpg, and one each of .pptx, .tex, .txt, .aux, .bib, .sty, .synctex, .log, .pygtex, .zip. These files contain saved HTTP error responses. The upstream tool that wrote them was not identified. None occurs in pdf, ps or the HTML family.

Source rows at most 64 bytes, including empty files, total 209,219 across 73,256 papers and 6,891,503 bytes. They include 44,203 TeX-extension files (11,416 exact %auto-ignore markers), 41,878 build artifacts, 20,993 00README.XXX directives, 6,552 Subversion metadata files, 2,146 empty bibliography environments and 1,442 MIME declarations. Sixty-four are NUL-filled, including the 42 Krita records.

The one-, two- and three-byte files were classified exhaustively by matching their digests against possible byte strings: 2,431 one-byte rows across 1,622 papers (1,603 newlines), 2,501 two-byte rows across 1,155 papers (549 double newlines), and 2,225 three-byte rows across 555 papers (276 triple newlines).

Nested archives

Nested archives are retained without recursive expansion. The filename-based set contains 661 files across 496 papers, 1,730,346,673 raw bytes and 580 distinct blobs. It uses .rar (279), .xz (123), .7z (87), .tar (85), .gz (41), .zip (19), .zst (eleven), .tgz (six), .lzma (five), .bz2 (four) and .z (one). It excludes 61 .jar files and other containers such as .sit, .egg, .deb, .cab, .lzh, .dmg, .pkg. Of the selected files, 151 are ancillary, containing 787,951,200 bytes (45.5% of selected archive bytes).

All 661 files were read from Parquet and their headers inspected. The audit found 555 nonzero containers, plus 68 entirely NUL-filled files and 38 other non-containers. The 68 contain 78,649,779 bytes (63 .tar, five .tgz); 52 of them are valid empty archives, so grouping all 106 residual files as “not archives” would be misleading. The other 38 are thirty xv thumbnails under xxx.xvpics/, three uncompressed TeX files, two PDFs named .zip, one AppleDouble sidecar, one PostScript file named .Z and one nine-byte Not Found body named .zip. All 106 together contain 83,788,775 bytes across 77 papers.

Of the 555 nonzero containers, eight differ from their filename format: five .zip files are RAR, one is 7z and two .tar files are xz.

Licensing

Paper licences vary. In the metadata population, 60.42% carry arXiv's non-exclusive distribution licence and 14.38% have no recorded licence. The non-exclusive licence grants rights to arXiv; it does not itself grant downstream redistribution rights. A missing value is not a public-domain declaration. See arXiv's licence terms and this repository's LICENSE.

The recorded paper-level licence is copied to versions; it is not a per-version licence history. arXiv notes that different revisions can have different licences. Check the relevant version where that distinction matters. arXiv separately releases its metadata under CC0, and this compilation's organisational layer is also dedicated to CC0. Neither statement relicenses the papers' full text. See the arXiv licensing policy.

Attribution

Cite the papers your work uses as well as the dataset. The supplied arxiv_abs_url and arxiv_pdf_url fields provide links to their arXiv records and downloads.

@misc{arxiv_complete_2026,
  title  = {arXiv Complete Corpus},
  author = {secemp9},
  year   = {2026},
  note   = {Snapshot: metadata 2026-08-30, files 2026-09-05},
  url    = {https://huggingface.co/datasets/secemp9/arxiv-complete}
}

Please also acknowledge arXiv as the source of the material. Use of its interfaces is governed by the arXiv API terms.

Maintenance

This release has no automatic refresh. A rebuild would update the snapshot dates and recompute the version flags.

Report errors, corrections or takedown requests through a discussion on this repository. Authors requesting removal will have their work removed from the next revision, as described in LICENSE.