Britannica Illustrated Pages

115,293 illustrated pages from scanned volumes of the Encyclopaedia Britannica, 1st edition (1768–71) to 14th (1929), selected by a page classifier from 975,345 pages in 1,160 volumes (838 Internet Archive items). A second config carries the classifier score, OCR word count and provenance for every one of the 975,345 pages.

Two things the scan showed:

• 82% of the illustrated pages are text pages (≥100 OCR words) — figures, diagrams and engravings set inside columns of text, not full-page plates. Illustration mining in digitised books usually filters for image-only pages first. That filter would have missed 94,762 of these 115,293 pages.
• The illustration rate quadruples across the run. 3rd–6th editions (1788–1824): 3.4–3.9% of pages illustrated. 9th–12th (1875–1923): 15–16%. Table below.

Examples

Each image links to the same page in the Internet Archive's viewer (ia_page_url).

Illustration rate by edition

Scored pages per edition and the share the classifier flagged as illustrated. Edition labels come from the IA catalogue record where it names one, else are inferred from the year (edition_inferred = true); 8% of pages have neither.

The 1st-edition rate (9.0%) is inflated by the NLS 1771 scans, where the copperplates are bound together and score at 24%; the 1773 printing scores 5%. The 14th-edition figure rests on 16 volumes. Copies of the same edition from different libraries are all counted — the volume counts are scanned volumes, not distinct volumes.

Configs

manifest is the one to re-threshold, compute rates per edition or decade, or build your own split from; pages is the one to browse or train on.

Columns

Both configs share:

pages adds image. manifest adds stored (whether the page made it into pages), src_archive, src_djvu, sha256, model, run_id, licenseurl, source_manifest.

tier and stratum disagree on ~10k pages. The tier was fixed when the page was stored; the stratum was recomputed from the correct word count after the page-order repair. Use stratum.

Method

Every page of every volume was decoded from the IA _jp2.zip / _images.zip masters and scored by davanstrien/britannica-illustrated-detector (timm efficientvit_b1.r256_in1k, 7.5M parameters) on Hugging Face Jobs — 16 cpu-performance shards reading the scans from an HF bucket, ~$3 total. The classifier itself was trained on Jobs (a10g-small, ~5 min). The classifier was trained on the National Library of Scotland's 2,573-page encyclopaedia_britannica_illustrated labels.

Thresholds were set on 347 human-labelled pages from this corpus, in two samples:

Recall on low-word pages is the weaker number: the misses were all near-threshold (0.91–0.98). Re-threshold from manifest if you want recall over precision.

Page order. For 242 volumes the image order inside the zip did not match the OCR order, so page numbers and word counts had been mis-bound. Those rows were re-bound via the zip central-directory permutation and verified against djvu page geometry (order_repaired = true on 640k rows, most of them no-ops). file_key keeps the original ordinal.

Provenance and licensing

All page images are scans held by the Internet Archive, contributed by libraries including the National Library of Scotland, University of California Libraries, Cornell, Harvard, the Getty Research Institute, the Bavarian State Library and the Digital Library of India — see contributor and ia_url per row. The volumes were published 1768–1929 and are public domain in the US by publication date. IA item-level licence fields vary (Public Domain Mark, CC-BY on some third-party uploads, blank); the underlying works are out of copyright regardless. Attribute the contributing library where one is named.

The IA "Britannica" search that seeded the mirror also returned items that are not pre-1930 encyclopaedia volumes. 8 were removed in v1.1 — two of them in copyright (Great Books of the Western World, 1952; Book of the Year 1938) that had passed a year-based screen because their IA records carry no date. The list with reasons is in curation/dropped_items.json. 2 of 1,162 volumes are missing (corrupt source archives).

Limitations

• Labels are classifier predictions. Precision was 1.000 on both validation samples, but the samples are small (347 pages) and drawn from the same corpus.
• Edition labels: 45% from the catalogue record, 46% inferred from year, 8% absent. The inference is year → edition and cannot separate later printings from first printings.
• year is null for 9% of pages (IA date blank or non-numeric — year_raw keeps the string).
• contributor is blank for a third of items.
• Same edition, several copies: 9th and 11th editions in particular are present from multiple libraries. Dedupe on edition + volume if you need one copy per volume.

Load

from datasets import load_dataset

# scores + provenance for every page, no images (~120 MB)
manifest = load_dataset("biglam/britannica-illustrated-pages", "manifest", split="train")

# the illustrated pages with images (~9.6 GB) — stream unless you want all of it
pages = load_dataset("biglam/britannica-illustrated-pages", "pages", split="train", streaming=True)
row = next(iter(pages))
row["image"], row["edition"], row["ia_page_url"]

Citation

Cite the Internet Archive item (ia_url) and contributing library for the scans, and the National Library of Scotland for the training labels. Classifier, scan and curation by Daniel van Strien (Machine Learning Librarian, Hugging Face); published under BigLAM, the BigScience/Hugging Face initiative for libraries, archives and museums data.

Page images (full resolution)

Every row links to its page in the public bucket biglam/britannica. This is the quickest way to get the full-resolution scan — no archive.org round-trip:

• bucket_url — the full-resolution original (source/pages/<ia_id>/<member>.jp2, straight from the Internet Archive item)
• bucket_url_jpg — a 2000 px JPEG of the same page (source/jpg/…), for browsers and quick loading
• leaf — the IA leaf number of the page; ia_page_url opens the same page on archive.org

The image column embedded in this dataset is a 600 px thumbnail. Every row was matched to its page by comparing that thumbnail with the bucket image (grey-level correlation ≥ 0.99) before the links were written; the match for every row is recorded in the bucket's manifest/rowmap-*.jsonl.

Crop masks (crop_masks config)

Instance boxes and masks for the illustrations on every page of this dataset, predicted by small-models-for-glam/britannica-illustration-detector-seg-v2 (RF-DETR Seg small, 29M parameters). One row per page, joinable on file_key; 115,293 rows, 700,480 instances at score ≥ 0.10, 411,385 at ≥ 0.30.

Each row carries objects (index-aligned lists: bbox as [x, y, w, h] px, score, area, rectangularity) and masks_rle (a JSON string of COCO RLE dicts in the same order). The frame is the full-resolution original (src_width × src_height, the image at bucket_url), not the embedded thumbnail; to use them on the 2000 px JPEG (bucket_url_jpg) scale by jpg_width / src_width.

Filter by score before use. Predictions are stored down to 0.10 so that consumers choose the operating point; 0.3 is a sensible display default (on a VLM-judged 78-page sample, instances between 0.25 and 0.30 were still ~97 % illustrations). Masks are the raw model output and sit slightly inside the drawn edges; for cut-outs, dilate by ~2 % of the box size:

import json
from datasets import load_dataset
from pycocotools import mask as maskutil
from scipy import ndimage

masks = load_dataset("biglam/britannica-illustrated-pages", "crop_masks", split="train")
row = masks[0]
rles = json.loads(row["masks_rle"])
keep = [i for i, s in enumerate(row["objects"]["score"]) if s >= 0.3]
m = maskutil.decode(rles[keep[0]])                       # H x W, full-resolution frame
x, y, w, h = row["objects"]["bbox"][keep[0]]
m = ndimage.binary_dilation(m, iterations=round(0.02 * max(w, h)))

Join against the page metadata with DuckDB, no image bytes fetched:

import duckdb
duckdb.sql("""
SELECT p.file_key, p.edition, p.year, p.bucket_url, m.objects, m.masks_rle
FROM 'hf://datasets/biglam/britannica-illustrated-pages/pages/*.parquet' p
JOIN 'hf://datasets/biglam/britannica-illustrated-pages/crop_masks/*.parquet' m USING (file_key)
""")

Masks are not clipped to their boxes: they normally sit inside the box, but low-score instances can extend beyond it — clip to bbox if you need the two to agree. Known limits: small in-text line diagrams are sometimes missed (about 6 % of illustrated pages had no detection at all on the judged sample); dense plates can merge neighbouring figures. Every row is stamped with model_id and model_sha.