ufal/atrium-nlp-enrich

NLP enrichment of text lines from ALTO XML files

stars

381

commits

Python

primary language

Sep 8, 2026

updated

named-entity-recognition

part-of-speech-tagging

README

📦 ALTO XML Files Postprocessing Pipeline - NLP Enrichment of text

This project provides a workflow for processing text stored in CSV (XLSX) with NLP services. It takes ordered text and extracts high-level linguistic features like Named Entities (NER) with tags and CONLL-U files with lemmas & part-of-sentence tags, and keywords (KER) per page/document.

[!CAUTION] This repository is a follow-up to main ALTO XML postprocessing GitHub repository, a part of ATRIUM project dedicated to ALTO-2-TXT workflow and collection of statistics and from text content of the documents (text and bounding boxes ordered by LayoutReader) recorder in CSV (XLSX) tables as a text column ¹.

TEITOK XML — Unified Output Format
⚙️ Setup
Workflow Stages
- Step 1: Prepare CSVs with texts from Page-Specific ALTOs
- Step 2: Extract NER and CONLL-U
  - Configuration ⚙️
  - Execution Pipeline
Output Structure
EXTRA: Extract Keywords (KER / YAKE / KeyBERT)
EXTRA: Converting Other Input Formats with flexiconv
EXTRA: LLM Semantic Enrichment (Vocabulary Mapping)
- Reviewing the vocabulary · runbooks: vocabulary · prompt & output
EXTRA: REST API Service
Paradata Logs
Acknowledgements

TEITOK XML — Unified Output Format

TEITOK XML (.teitok.xml) is the primary enriched output format of this pipeline. It is a TEI-compliant XML format used by the TEITOK corpus platform, extended to carry spatially-grounded linguistic and NER annotations produced by UDPipe and NameTag.

Each document in the collection is serialised as a single .teitok.xml file that integrates four layers of information in a consistent, machine-readable structure:

Layer	Content
Layout	Page, text-block, and line boundaries with pixel-accurate bounding boxes from the source ALTO XML, scaled to match the stored PNG images
Morphology & Syntax	Per-token lemma, UPOS/XPOS tags, morphological features, and dependency relations produced by UDPipe 2
Named Entities	BIO-tagged entity spans with both a CoNLL-style category (`PER`, `ORG`, `LOC`, `MISC`) and a fine-grained CNEC 2.0 code (e.g. `pf` = first name, `gu` = city) produced by NameTag 3
Facsimile links	`<surface>` elements in `<facsimile>` that tie each page to its companion image, enabling TEITOK's side-by-side text/image view

Why TEITOK XML?

Storing all enrichment layers in a single interoperable format offers several practical advantages over keeping CoNLL-U, TSV, and image files in separate silos:

🔍 Full-text and attribute search — TEITOK's built-in CQL/XPATH query engine lets users search across lemmas, NER types, POS tags, and raw text simultaneously.
🏷 Named entity access — entity spans (<name type="PER" cnec="pf">) are first-class XML elements: queryable, stylable, and exportable independently of the surrounding tokens.
🖱 Mouseover information — hovering over any token in the TEITOK GUI surfaces its lemma, morphological features, and dependency relation without leaving the page view.
🖼 Page visualisation with spatial overlays — bounding box coordinates on every <tok>, <lb>, and <div> are used by TEITOK's facsimile viewer to overlay text highlights directly onto the scanned page image, making OCR quality immediately visible.
📐 Layout-aware structure — text blocks (<div type="MarginTextZone-P">), lines (<lb>), and graphical elements (<figure>) preserve the physical layout of the original document.
🔗 Interoperability — TEI/XML is a widely adopted standard in digital humanities; the files can be ingested by other TEI-aware tools (e.g. eXist-db, Oxygen XML Editor) without conversion.

TEITOK XML structure at a glance

<TEI xmlns="http://www.tei-c.org/ns/1.0" xml:lang="cs">
  <teiHeader> ... </teiHeader>

  <facsimile>
    <surface id="doc1.surface1" lrx="1240" lry="1754">
      <graphic url="doc1-1.png"/>
    </surface>
  </facsimile>

  <text><body>
    <pb n="1" id="doc1.pb1" facs="doc1-1.png"/>

    <div type="MarginTextZone-P" id="doc1.TB_1" bbox="142 210 1098 880">
      <s id="doc1.s1" text="Výroční zpráva 2012 .">
        <lb id="doc1.TL_1" bbox="142 210 680 255"/>

        <tok id="doc1.s1.w1" type="w" lemma="výroční" upos="ADJ"
             feats="Case=Nom|..." deprel="amod"
             bbox="142 210 310 255">Výroční</tok>

        <name type="ORG" cnec="if">
          <tok id="doc1.s1.w3" type="w" lemma="ministerstvo" upos="NOUN"
               bbox="320 210 580 255">Ministerstvo</tok>
          <tok id="doc1.s1.w4" type="w" lemma="finance" upos="NOUN"
               bbox="585 210 680 255">financí</tok>
        </n>
      </s>
    </div>
  </body></text>
</TEI>

[!NOTE] TEITOK XML is generated by Step 4 of this pipeline (api_4_stats.sh) when SAVE_TEITOK=true. The source ALTO XML files must be present in INPUT_ALTO_DIR for spatial coordinates to be included. If INPUT_ALTO_DIR is not set, TEITOK XML is still produced but without bounding box attributes. If your documents are not in ALTO format, see EXTRA: Converting Other Input Formats with flexiconv.

⚙️ Setup

Before you begin, set up your environment.

Create and activate a new virtual environment in the project directory 🖥.
Install the required Python packages:

pip install -r requirements.txt

For keyword extraction, install the backend(s) you intend to use:

# YAKE — unsupervised statistical extraction, CPU-only
pip install yake

# KeyBERT — embedding-based extraction, GPU-accelerated when available
pip install keybert sentence-transformers
pip install torch          # optional — enables CUDA GPU acceleration

The original legacy KER backend requires no additional packages. For the LLM Semantic Enrichment pipeline, install the inference backend you intend to use:

# Transformers backend — single GPU, models ≤ 31 B (BnB 4-bit / AWQ / GGUF)
pip install -r requirements_llm.txt

# vLLM backend — multi-GPU, large models (≥ 70 B), Automatic Prefix Caching
# Replaces lmformatenforcer; uses xgrammar for native guided JSON decoding
pip install vllm

(Optional) To run the REST API service, install additional requirements:

pip install -r service/requirements.txt

Review and update the config_api.txt 📎 file with your specific paths and API configurations. You are now ready to start the workflow.

Workflow Stages

The process is divided into sequential steps, each responsible for a specific part of the NLP enrichment pipeline.

▶ Step 1: Prepare CSVs with texts from Page-Specific ALTOs

[!IMPORTANT] If you already have a directory of CSV (XLSX) tables with text column containing extracted text files from ALTO XMLs, you can skip Step 1 and proceed directly to Step 2.

The ../CSVS_with_TEXT/ directory mentioned later is the result of ALTO XML postprocessing pipeline described in the separate repository ¹. It contains document-specific CSV (XLSX) files with the text column containing extracted textual content from the ALTO XML files. Each CSV (XLSX) file corresponds to a document and contains rows for each page with a line number column for the proper ordering (page_num and line_num).

CSVS_with_TEXT/
├── document1.csv
├── document2.csv
└── ...

with the structure of each CSV (XLSX) file like:

file,page_num,line_num,text,split_ws,split_we,lang,lang_score,perplex,categ
CTX201504033,1,8,2012,,,N/A,0,0,Non-text
CTX201504033,2,2,1,,,N/A,0,0,Non-text
CTX201504033,3,2,2,,,N/A,0,0,Non-text
...

Where split_ws and split_we are the start and end character offsets of the words split in the original ALTO XML. The lang and lang_score columns indicate the detected language and its confidence score, while perplex and categ provide additional metadata about the text classification.

If the script detects an .xlsx file, it will iterate over all sheet names, verify if a text column exists in each sheet, and extract the content safely for Excel tables with multiple sheets.

▶ Step 2: Extract NER and CONLL-U

This stage performs advanced NLP analysis using external APIs (Lindat/CLARIAH-CZ) to generate Universal Dependencies (CoNLL-U) and Named Entity Recognition (NER) data.

Unlike previous steps, this process is split into modular shell scripts to handle large-scale processing, text chunking, and API rate limiting.

Configuration ⚙️

Before running the pipeline, review the api_config.txt 📎 file. This file controls directory paths, API endpoints, and model selection.

# config_api.txt
OUTPUT_DIR="../../ARUB"                          # Destination for results
INPUT_TABLES_DIR="$OUTPUT_DIR/DOC_LINE_LR_CLS"  # Input tables from Step 1

WORK_DIR="./TEMP"                                # Working directory for intermediate files

LOG_FILE="$OUTPUT_DIR/processing.log"
CONLLU_INPUT_DIR="$OUTPUT_DIR/UDP"
TEMP_TXT_DIR="./TEMP/TXT_EXTRACT"
CHUNK_DIR="./TEMP/CHUNKS"

TSV_INPUT_DIR="$OUTPUT_DIR/NE"
SUMMARY_OUTPUT_DIR="$OUTPUT_DIR/UDP_NE"

TEITOK_OUTPUT_DIR="$OUTPUT_DIR/TEITOK"
INPUT_ALTO_DIR="$OUTPUT_DIR/altos"              # Source ALTO XML files - for TEITOK conversion
# ── Image Options ─────────────────────────────────────────────────────────────
# OPTIONAL: Only required if your companion PNG/JPEG display images have been resized
# to a different target resolution relative to ABBYY's baseline dimensions.
# If left empty, the pipeline calibrates layout shifts natively using ALTO PrintSpace.
INPUT_PAGES_DIR=""

UDPIPE_URL="https://lindat.mff.cuni.cz/services/udpipe/api/process"
NAMETAG_URL="https://lindat.mff.cuni.cz/services/nametag/api/recognize"

MODEL_UDPIPE="czech-pdt-ud-2.15-241121"
MODEL_NAMETAG="nametag3-czech-cnec2.0-240830"

TIMEOUT=60                     # API call timeout in seconds
MAX_RETRIES=5                  # Number of retries for failed API calls
BACKOFF_FACTOR=1.5
WORD_CHUNK_LIMIT=900           # Word limit per API call

SAVE_CSV=true                  # write token-level summary CSV
SAVE_CONLLU_NE=true            # keep merged CoNLL-U with NER in MISC
SAVE_TEITOK=true               # write TEITOK-style TEI XML (flexiconv-compatible)

Execution Pipeline

Run the following scripts in sequence. Each script sources config_api.txt 📎 directly for configuration. Retry logic and per-attempt error handling are implemented inside the Python helper scripts (call_udpipe.py, call_nametag.py) using exponential back-off controlled by the MAX_RETRIES and BACKOFF_FACTOR variables. api_util/api_common.sh 📎 is a standalone utility module that exposes a log() helper and an api_call_with_retry() shell function for any custom scripts that choose to source it; the four main pipeline scripts (api_1_manifest.sh … api_4_stats.sh) do not source it. Additionally, api_util/ 📁 contains helper Python scripts for chunking and analysis

1. Generate Manifest

Maps input text files to document IDs and page numbers to ensure correct processing order.

./api_1_manifest.sh

Input: ../CSVS_with_TEXT/ (raw text files in subdirectories from Step 1).
Output: OUTPUT_DIR/manifest.tsv.

Example output file manifest.tsv 📎 with file, page number, and path columns. It lists all text files to be processed in the next steps. Run the following command to see how many documents will be processed:

tail -n +2 OUTPUT_DIR/manifest.tsv | wc -l

which returns the total number of document rows in the manifest, excluding the header line.

2. UDPipe Processing (Morphology & Syntax)

Sends text to the UDPipe API ². Large documents are automatically split into chunks (default 900 words) using chunk.py 📎 to respect API limits, then merged back into valid CoNLL-U files.

./api_2_udp.sh

Input 1: OUTPUT_DIR/manifest.tsv (mapping of text files to document IDs and page numbers).
Input 2: ../CSVS_with_TEXT/ (raw text files in subdirectories from Step 1).
Output: OUTPUT_DIR/UDP/*.conllu (Intermediate per-document CoNLL-U files).

Run the following command to see how many documents have been processed into CoNLL-U files:

ls -l <OUTPUT_DIR>/UDP/ | wc -l

which returns the total number of CoNLL-U files created (each file corresponds to a document).

Example output directory UDP 📁 contains per-document CoNLL-U files.

[!NOTE] Chunking and page boundaries. chunk.py📎 splits text on OCR line boundaries (not raw whitespace), preserving the newline-separated structure of the source CSV so that UDPipe receives proper sentence-boundary hints between lines. When a document spans multiple chunks, call_udpipe.py📎 merges them into a single CoNLL-U and injects a # page_break = true comment immediately before every sentence that began a new page in its source chunk. All downstream scripts (call_nametag.py📎, summarize_nt_udp.py📎, teitok_alto.py📎) recognise this marker alongside the legacy # sent_id = 1 page-reset convention, so both single-chunk and multi-chunk files are handled transparently.

[!TIP] You can launch the next step when a portion of CoNLL-U files are ready, without waiting for the entire input collection to finish. You will have to relaunch the next step after all CoNLL-U files are ready to process the files created after the previous run began.

3. NameTag Processing (NER tags)

Takes the valid CoNLL-U files and passes them through the NameTag API ³ to annotate Named Entities (NE) directly into the syntax trees.

./api_3_nt.sh

Input: OUTPUT_DIR/UDP/*.conllu (Intermediate per-document CoNLL-U files).
Output: OUTPUT_DIR/NE/*/*.tsv (NE annotated per-page files)

Run the following command to see how many documents have been processed into TSV files:

ls -l OUTPUT_DIR/NE | wc -l

which returns the total number of directories created (each subfolder corresponds to a document).

Example output directory NE 📁 contains per-page TSV files with NE annotations, where the NE tags follow the CNEC 2.0 standard ⁴ which is used in the Czech Nametag model.

4. Generate Statistics

This stage consolidates the linguistic data from UDPipe (CoNLL-U) and the NER data from NameTag (TSV) into final per-document formats. It also generates a master summary of entity counts across the entire collection and can optionally produce TEITOK-compatible XML files that merge linguistic tokens with original ALTO layout coordinates.

The process utilizes summarize_nt_udp.py 📎 to merge these layers, map complex CNEC 2.0 tags (e.g., g, pf, if) into human-readable categories (e.g., "Geographical name", "First name", "Company/Firm"), and write all output formats. Optionally, TEITOK-related functionality is implemented in teitok_alto.py 📎.

./api_4_stats.sh

Inputs and Outputs

Input 1: OUTPUT_DIR/UDP/*.conllu — Per-document CoNLL-U files containing morphology and syntax.
Input 2: OUTPUT_DIR/NE/*/*.tsv — Per-page TSV files containing Named Entity annotations.
Input 3 (Optional): INPUT_ALTO_DIR/*.alto.xml — Source ALTO XML files used during TEITOK conversion to provide spatial bounding box coordinates for each token.
Input 4 (Optional): INPUT_PAGES_DIR/<doc_id>-N.png — Per-page facsimile images. When specified, the pipeline dynamically extracts pixel boundaries from the headers to compute scaling transformations (sx, sy). If omitted, coordinates are safely translated and aligned at a native 1.0 scale factor.
Output 1: OUTPUT_DIR/summary_ne_counts.csv — Global table of aggregated Named Entity statistics across all documents.
Output 2: OUTPUT_DIR/UDP_NE/<doc_id>/<doc_id>.csv — Per-document CSV tables with tokens, lemmas, and human-readable NE explanations.
Output 3 (Optional): OUTPUT_DIR/UDP_NE/<doc_id>/<doc_id>.conllu — Final CoNLL-U files with NER tags enriched in the MISC column.
Output 4 (Optional): OUTPUT_DIR/TEITOK/<doc_id>.teitok.xml — TEITOK-style TEI XML files ready for the flexiconv converter and facsimile viewing (see below).

The behavior of this step is controlled by boolean flags in your config_api.txt:

Variable	Description	Default
`SAVE_CONLLU_NE`	Keep the enriched CoNLL-U with NER in the `MISC` field.	`true`
`SAVE_CSV`	Write the token-level summary CSV per document.	`true`
`SAVE_TEITOK`	Write TEITOK-style TEI XML with bounding boxes and NER spans. When `INPUT_ALTO_DIR` is not set a warning is emitted and TEITOK XML is still produced without bboxes. If `INPUT_ALTO_DIR` is set but the path does not exist, the step exits with an error.	`true`
`INPUT_PAGES_DIR`	Directory of per-page images (`<doc_id>-N.png`). When set, bbox coordinates are scaled to match the actual PNG resolution. Leave empty to write raw ALTO pixel values.	(empty)

ALTO-to-TEITOK XML Generation and Coordinate Alignment

When SAVE_TEITOK=true, teitok_alto.py 📎 reads and processes the internal spatial hierarchy of your ALTO source specifications.

Offset Alignment (Resolving Layout Shifting):

ABBYY FineReader naturally indexes element positions from the absolute physical boundary of the scanner bed (0,0). However, companion web images cropped for public view or optimized to strip away raw scanner artifacts introduce a uniform positional drift (causing text layers to display too far left or too high up on screen).

To neutralize this error without modifying binary assets or re-cropping, the script automatically parses page-level <PrintSpace> properties from the ALTO structure:

<PrintSpace HEIGHT="3263" WIDTH="2027" VPOS="80" HPOS="297">

The horizontal boundary (HPOS) and vertical boundary (VPOS) values are captured as active translation variables (dx, dy). Prior to rendering bounding boxes into the TEITOK XML stream, these values are subtracted from the coordinate targets, recalculating alignment automatically:

$$\text{Scaled Coordinate} = \text{round}((\text{Absolute Coordinate} - \text{Offset}) \times \text{Scale Factor})$$

Dynamic Scale Calculations:

Companion Image Present (Tier 1): If INPUT_PAGES_DIR is set and matching images exist, the tool safely reads binary file headers without invoking bloated third-party imaging dependencies. Ratios are resolved by evaluating layout sizes against image shapes (sx = img_width / alto_width).
User-set DPI (Tier 2): If no image is available, scale is derived directly from the ALTO <MeasurementUnit> (inch1200, mm10, or pixel) mapped against the environment variables IMAGE_DPI and ALTO_DPI.
Native Processing (Fallback): If no image and no DPI is provided, the tool calibrates positions through native PrintSpace logic but maintains a standard 1.0 scale factor.

Future direction: Relative / resolution-independent coordinates are the preferred long-term direction (pending TEITOK-team confirmation).

Fixing Bounding Box Alignments (Practical Guide)

If you are a new user approaching this pipeline—perhaps a researcher who just digitized a batch of archival documents—your primary goal might be making sure the semantic annotations actually line up with your page images in a web viewer.

Let's say your original document was processed at a massive archival resolution, but the image you are serving to your web frontend is exactly 1200 pixels wide and 1800 pixels high. Currently, your TEITOK XML bounding boxes are completely misaligned.

Here is exactly how you would use the integrated tools to solve this problem.

Method 1: The Quick API Fix (Best for single files or web integrations)

Since the pipeline now includes a dedicated FastAPI service, you don't even need to write a script. You can just send your misaligned XML to the /rescale endpoint.

Open your terminal and run a simple curl command, explicitly telling the API the exact dimensions of your target image and requesting the output as an XML file instead of the default JSON metadata:

curl -X POST "http://localhost:8000/rescale" \
     -F "file=@CTX000000001.teitok.xml" \
     -F "width=1200" \
     -F "height=1800" \
     -F "format=xml" \
     -o CTX000000001.rescaled.teitok.xml

What happens behind the scenes: The API automatically detects the original coordinate space from the <surface> tag in your XML. It calculates the exact scaling factors needed to stretch or shrink the bounding boxes (bbox) to fit the new 1200x1800 dimensions. As a bonus, it also silently repairs any malformed named-entity tags (like <name>...</n>) in the document.

Method 2: The Command-Line Batch Process (Best for whole directories)

If you have hundreds of XML files in a folder and you know exactly what scale ratio or DPI conversion you need, using the REST API file-by-file would be tedious. Instead, use the dedicated CLI tool, fix_teitok_bboxes.py.

If you know your web images are exactly 50% the size of your original scans (a scale factor of 0.5), you can process the entire directory at once:

python3 fix_teitok_bboxes.py -i /path/to/my/teitok_folder/ --sx 0.5 --sy 0.5

Alternatively, if your original ALTO OCR data was in millimeters (mm10) and you need to target a standard 72 DPI screen resolution, the script can handle that math directly:

python3 fix_teitok_bboxes.py -i my_document.teitok.xml --unit mm10 --dpi 72

If the original scans included a scanner bed margin (e.g., 50 pixels on the left and 20 on the top) that was cropped out of the final web image, you can strip that out by shifting everything left and up:

python3 fix_teitok_bboxes.py -i my_document.teitok.xml --dx -50 --dy -20

Both methods directly address the historical pain point of facsimile alignment, allowing you to flawlessly overlay the NLP enrichments onto the visual documents without needing to re-run the entire pipeline.

[!NOTE] When a token's matched ALTO strings span more than one page (a rare OCR edge case near page boundaries), a warning is printed to stderr identifying the token and the conflicting page indices. The first matched page is used for the bbox assignment in that case.

The structural and spatial hierarchy from the ALTO file is strictly preserved in the generated TEITOK XML:

Tokens: Matched coordinates are written to each <tok> element as @bbox="x1 y1 x2 y2" (absolute pixel coordinates in TEITOK's hOCR-derived format). Each token also carries @type="w" (word) or @type="pc" (punctuation character) derived from UDPipe's UPOS tag.
Lines: ALTO <TextLine> elements are preserved via <lb> (line break) tags, which also include their own @bbox spatial coordinates.
Blocks: Text blocks are encapsulated within <div type="MarginTextZone-P"> containers, satisfying the core ATRIUM guidelines for classified text zones.
Graphics: Non-text elements like Illustration and GraphicalElement blocks are parsed and appended to their respective pages as <figure> tags with strict bounding boxes.
Pages: Page boundaries are marked with <pb n="N" id="..." facs="..."/> elements pointing to the specific document surface.

Named entity spans are wrapped in <n> elements grouping their constituent <tok> nodes. Two attributes encode the entity type at different levels of granularity: @type holds the CoNLL-style category (PER, ORG, LOC, or MISC) intended for querying and interoperability, while @cnec carries the raw CNEC 2.0 code (e.g., pf, gu, if) for use in visualisation. For example, a span tagged as a first name is written as <name type="PER" cnec="pf">.

[!NOTE] Thanks to the sequence matching approach, the script achieves near-perfect spatial alignment between NLP tokens and OCR coordinates, drastically improving upon older greedy matching methods that would break on minor character variations. Alignment statistics (matched vs. total tokens) are printed to the console per document.

ls OUTPUT_DIR/UDP_NE | wc -l

which returns the total number of created files, both .csv and .conllu corresponding to specific documents.

ls OUTPUT_DIR/UDP_NE/*/*.csv | wc -l

returns number of documents processed into tables

ls OUTPUT_DIR/TEITOK/*.xml | wc -l

returns number of recorded .teitok.xml documents.

Example summary table: summary_ne_counts.csv (produced by a real run; not committed — see the note below).

Example output directory UDP_NE 📁 contains per-document CSV tables with NE tags and UDPipe feature columns, plus CoNLL-U files with NE annotations in per-document manner.

Example output directory TEITOK 📁 contains per-document TEITOK XML files combining UD linguistic annotations and NER spans with bounding boxes aligned from the source ALTO XML.

Output Structure

After completing the pipeline, your working and output directories will be organized as follows:

TEMP/
├── CHUNKS/
│   └── ...
├── nametag_response_docname1.conllu.json
└── ...

AND

<OUTPUT_DIR>
├── UDP_NE/
│   ├── <doc_id>
│   │   ├── <doc_id>.csv
│   │   └── <doc_id>.conllu
│   ├── <doc_id>
│   │   ├── <doc_id>.csv
│   │   └── <doc_id>.conllu
│   └── ...
├── UDP/
│   ├── <doc_id>.conllu
│   ├── <doc_id>.conllu
│   └── ...
├── TEITOK/
│   ├── <doc_id>.teitok.xml
│   ├── <doc_id>.teitok.xml
│   └── ...
├── NE/
│   ├── <doc_id>
│   │   ├── <doc_id>-<page_num>.tsv
│   │   └── ...
│   ├── <doc_id>
│   │   ├── <doc_id>-<page_num>.tsv
│   │   └── ...
│   └── ...
├── altos/
│   ├── <doc_id>.alto.xml
│   └── ...
├── pages/
│   ├── <doc_id>-1.png
│   ├── <doc_id>-2.png
│   └── ...
├── processing.log
├── summary_ne_counts.csv
└── manifest.tsv

The combined output summary_ne_counts.csv contains aggregated Named Entity statistics across all processed pages. This repository's data_samples/ only ships the three synthetic demo documents (CTX00000000{1,2,3}), so no summary_ne_counts.csv is committed — the file is real output of a real api_5_summary_ne.sh run, not a sample bundled here.

[!NOTE] Now you can delete UDP/ from <OUTPUT_DIR>/ if you no longer need the raw CoNLL-U files. The final CoNLL-U files with NER features are in <OUTPUT_DIR>/UDP_NE/.

If you do not plan to rerun any part of the pipeline, you can also delete the entire TEMP/ directory including manifest.tsv 📎.

EXTRA: Extract Keywords (KER / YAKE / KeyBERT)

[!NOTE] This is an optional step in NLP enrichment of your data. It can give a fast thematic overview of the whole collection and works best when UDPipe lemmas (output of Step 2) are available. Three extraction backends are provided; choose the one that best fits your environment and quality requirements.

Extract keywords 🔎 from your documents by running keywords.py on a directory of CoNLL-U files produced by Step 2.

Configuration Priority

The keyword extraction script uses a three-tier configuration hierarchy (from highest to lowest priority):

Command-line flags (e.g., -m yake, -w 3) always override everything else.
kw_config.txt (the [DEFAULTS] section) is read automatically if placed next to the script.
Hardcoded fallbacks are used if no config file or flags are provided.

This means if you configure your settings in kw_config.txt, you can simply run:

python3 keywords.py

Backends

Flag value	Method	Dependencies	Score semantics	Best for
`legacy`	Original KER — NOUN/PROPN/ADJ lemma frequency	none (stdlib only)	raw occurrence count	reproducing original ATRIUM results
`yake` (default)	YAKE — unsupervised statistical, CPU-only	`pip install yake`	normalised inverse YAKE score, [0, 1]	fast CPU runs, no model download
`keybert`	KeyBERT — embedding-based, GPU-accelerated	`pip install keybert sentence-transformers`	cosine similarity, [0, 1]	highest semantic quality, GPU recommended

You can override any kw_config.txt setting via the command line:

python3 keywords.py -i <input_dir> -m <method> -l <lang> -w <integer> \
                    -n <integer> -d <output_dir> -o <output_file>.csv

All available flags:

Flag	Long form	Default in `kw_config.txt`	Description
`-i`	`--input_dir`	`data_samples/UDP`	CoNLL-U directory to process
`-m`	`--method`	`yake`	Backend: `legacy`, `yake`, or `keybert`
`-l`	`--lang`	`cs`	Language code for YAKE stopwords (`cs`, `en`, `de`, …). Ignored by `legacy` and `keybert`
`-w`	`--max_words`	`3`	Maximum words per keyword phrase (n-gram upper bound)
`-n`	`--num_keywords`	`20`	Number of keywords to extract per document
`-d`	`--per_doc_out_dir`	`data_samples/KW_PER_DOC`	Output directory for per-document CSV files
`-o`	`--output_file`	`keywords_summary.csv`	Master keywords CSV
	`--keybert-model`	`paraphrase-multilingual-MiniLM-L12-v2`	Sentence-Transformer model name (KeyBERT only)
	`--no-mmr`	(False)	Disable Maximal Marginal Relevance diversification (KeyBERT only)
	`--diversity`	`0.5`	MMR diversity parameter, 0 = max relevance → 1 = max diversity (KeyBERT only)
	`--workers`	`0` (Auto / CPU count)	Parallel worker processes. Auto-forced to 1 for KeyBERT + GPU

Examples:

YAKE — Czech, up to 3-word phrases, 20 keywords per document (default)

python3 keywords.py -i OUTPUT_DIR/UDP -m yake -l cs -w 3 -n 20 \
        -o keywords_summary.csv -d KW_PER_DOC

KeyBERT — multilingual model, GPU-accelerated

python3 keywords.py -i OUTPUT_DIR/UDP -m keybert -w 3 -n 20 \
        --keybert-model paraphrase-multilingual-MiniLM-L12-v2 \
        -o keywords_summary.csv -d KW_PER_DOC

Legacy KER — (English/Czech) original ATRIUM lemma-frequency approach, no extra dependencies

python3 keywords.py -i OUTPUT_DIR/UDP -m legacy -n 20 \
        -o keywords_summary.csv -d KW_PER_DOC

[!WARNING] For KeyBERT with a GPU, the script automatically forces --workers 1 to prevent competing CUDA context initialisation across subprocesses. On CPU, any worker count is safe.

Inputs and outputs

Input: Directory of per-document CoNLL-U files from Step 2.
Output 1: Master table with keywords per document (e.g., keywords_summary.csv).
Output 2: Per-document CSV files (e.g., KW_PER_DOC/).

KW_PER_DOC/
├── <docname1>_keywords.csv
├── <docname2>_keywords.csv
└── ...

Each per-document file contains two columns — keyword and score — sorted by score in descending order. The master summary uses the same column structure as the original pipeline (document_id, kw-1, score-1, kw-2, score-2, …).

Score interpretation by backend

legacy — raw lemma count; higher = more frequent in the document. Examples in directory: KW_PER_DOC_L 📂 and summary file kw_summary_l.csv 📎.

Score range	Interpretation
1–5	Common functional nouns, low informativeness
5–20	Topic-representative vocabulary
> 20	Dominant terms, likely named entities or domain headings

yake — normalised inverse YAKE score, [0, 1] per document. Examples in directory: KW_PER_DOC_Y 📂 and summary file kw_summary_y.csv 📎.

Score range	Semantic category	Interpretation
0.0–0.2	Noise floor	Common words, low local relevance
0.2–0.6	Context layer	General vocabulary defining the broad topic
0.6–0.9	Topic layer	Specific nouns and verbs central to the text
0.9–1.0	Entity layer	Rare terms, neologisms, named entities

keybert — cosine similarity to document centroid, [0, 1]. Examples in directory: KW_PER_DOC_KB 📂 and summary file kw_summary_kb.csv 📎.

Score range	Interpretation
< 0.3	Weakly related phrases
0.3–0.6	Contextually relevant terms
> 0.6	Highly representative keyphrases

EXTRA: Converting Other Input Formats with flexiconv

[!NOTE] This section is relevant when your documents originate from an OCR or digitisation pipeline that does not produce ALTO XML — for example, PAGE XML, hOCR, plain-text exports, or proprietary formats. If you already have ALTO XML, the pipeline generates TEITOK XML natively via api_4_stats.sh (see above).

What is flexiconv?

flexiconv ⁵(https://github.com/ufal/flexiconv) is a flexible format-conversion tool developed at UFAL that translates a variety of OCR and document layout formats into TEITOK XML — the unified output format used by this project. It acts as a universal adapter: once your documents are in TEITOK XML, they can be ingested directly into the TEITOK corpus platform and will benefit from all the same search, visualisation, and NER capabilities described above.

  Your input format           flexiconv             Unified output
  ─────────────────    ─────────────────────────   ─────────────────
  PAGE XML          ─┐
  hOCR              ─┤──► flexiconv ──────────────► .teitok.xml ──► TEITOK platform
  plain text + CSV  ─┤                                            ──► this pipeline
  other OCR output  ─┘                                               (NER, KWs, ...)

When to use flexiconv

Use flexiconv before running this pipeline when:

Your collection was OCR-processed with a tool that outputs PAGE XML (e.g. Transkribus, OCRopus, kraken).
Your layout data is in hOCR format (used by Tesseract and some ABBYY exports).
You have structured text with positional metadata but no standard bounding-box format.
You received digitised material from a partner institution using a format not natively supported by teitok_alto.py.

How to use flexiconv

Clone and install the tool:

git clone [https://github.com/ufal/flexiconv.git](https://github.com/ufal/flexiconv.git)
cd flexiconv
pip install -r requirements.txt

Run the conversion on your input files:

python flexiconv.py \
    --input-dir  /path/to/your/source/documents \
    --input-fmt  page-xml \          # or: hocr, plain, ...
    --output-dir /path/to/teitok_out \
    --output-fmt teitok

Refer to the flexiconv documentation for the full list of supported --input-fmt values and format-specific options.

Continue with this pipeline using the converted TEITOK XML files. At this point your documents already have layout structure and bounding boxes embedded — the NLP enrichment steps (UDPipe morphology, NameTag NER, keyword extraction) can be applied on top via the scripts in this repository.

[!TIP] If your format is not yet supported by flexiconv, please open an issue on the flexiconv GitHub repository. The tool is actively developed within the ATRIUM project and new format adapters are added regularly.

EXTRA: LLM Semantic Enrichment (Vocabulary Mapping)

[!NOTE] This is an advanced, optional step. It runs a local Large Language Model to semantically analyse each text line and map it to the controlled TEATER/AMCR archaeological vocabulary. Two inference backends are supported: transformers (HuggingFace + BnB 4-bit, single GPU, models ≤ 31 B) and vllm (multi-GPU, Automatic Prefix Caching, native guided JSON decoding, models ≥ 70 B or any multi-GPU node).

This pipeline goes beyond traditional keyword extraction by using Constrained Decoding. For the transformers backend this is implemented via Pydantic schemas and lmformatenforcer. For the vllm backend, guided decoding is handled natively by xgrammar inside vLLM — no additional library is required. In both cases the model is mathematically prevented from producing any token that would violate the predefined JSON structure or select a vocabulary term outside the thematic dictionary, entirely eliminating hallucinated formatting.

⚙️ Configuration (llm_config.txt 📎)

The pipeline reads all runtime parameters from llm_config.txt in the repository root. The minimum required change is MODEL_KEY; every other key has a sensible default.

# Single-GPU (BACKEND=transformers): qwen-3.6-27b-it | gemma-4-31b-it | qwen3-14b |
#                                    qwen-3.5-9b-it | qwen3-8b | qwen2.5-14b-awq |
#                                    qwen2.5-7b | gemma-3-12b-it
# MoE / GGUF (single GPU):           gemma-4-26b-moe-gguf | qwen-3.6-35b-moe
# Multi-GPU (BACKEND=vllm):          qwen3-235b-a22b-fp8 | deepseek-v3 | llama4-maverick | llama3.1-70b
MODEL_KEY=qwen-3.6-27b-it

# Only needed for gated models: gemma-4-*, llama4-maverick, llama3.1-70b
# HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxx

INPUT_DIR=data_samples/DOC_LINE_CATEG
OUTPUT_DIR=data_samples/KW_PER_DOC_LLM
VOCAB_PATH=data_samples/vocab/union_nested.json
PARADATA_DIR=paradata

# Attach the surviving vocabulary term's source record id(s) to each enrichment as
# teater_category_ids (issue #6, M7). Kept behind a switch since it was agreed to be
# reversible: "list them now and drop it if it will create some issues."
EMIT_CATEGORY_IDS=true

INCLUDE_NON_TEXT=true
MIN_CHAR_COUNT=3
MIN_CHAR_NON_TEXT=8
MIN_ALPHA_RATIO_NON_TEXT=0.4

# ── System prompt ─────────────────────────────────────────────────────────────
# The instruction text lives in prompts/system_prompt.txt as [[named blocks]]; these
# flags choose which of them render. The run banner prints the resulting on/off list,
# so a log always says what the model was told. See prompts/RUNBOOK.md.
PROMPT_TEMPLATE=prompts/system_prompt.txt
PROMPT_TASK_EXTRACT=true
PROMPT_TASK_SELECT=true
PROMPT_METATEXT_RULE=true
PROMPT_OCR_NORMALISATION=true
PROMPT_EXACT_TERM=true
PROMPT_EXAMPLES=true

# The one three-way switch: strict | preference | off. Paired with
# taxonomy_config.json's geo_guardrail.active — vocab_build.py refuses a build where
# the two disagree.
PROMPT_GEO_GUARDRAIL=preference

# Term-list layout: facet_sub | facet | flat. Same terms, same truncation; only the
# headers move.
PROMPT_VOCAB_GROUPING=facet_sub

# ── Inference parameter overrides ─────────────────────────────────────────────
# ALL of these are COMMENTED OUT in the shipped file. Backend, GPU count, memory
# utilisation, batch size and context cap are resolved automatically from the model
# registry in llm_utils.py; the startup log prints every effective value next to
# where it came from (← llm_config.txt / model default / global default). Uncomment
# a line only to deviate from the model's recommended configuration.
# BACKEND=vllm                 # transformers (single GPU, ≤ 31 B) | vllm (multi-GPU)
# TENSOR_PARALLEL_SIZE=8       # GPUs to shard across (vLLM only)
# GPU_MEMORY_UTILIZATION=0.88  # Fraction of each GPU's VRAM for the KV cache
# VLLM_BATCH_SIZE=8            # Lines per generate() call
# MAX_MODEL_LEN=16384          # Cap the context window to reduce KV-cache pressure
# CPU_OFFLOAD_GB=0             # Weights to keep in CPU RAM when VRAM is short
# GUIDED_DECODING_BACKEND=xgrammar
# ENABLE_PREFIX_CACHING=false  # Not recommended; reduces throughput

The nine PROMPT_* keys are the prompt's whole configuration surface; prompts/RUNBOOK.md 📎 documents each block, what it costs, and which pairings are unsafe to change alone.

🗂 Workflow

1. Vocabulary Harvesting (vocab_build.py 📎)

The vocabulary is built in two stages, and only the first needs the internet:

harvest (network)   →   FLAT artifacts    →   nest (pure)    →   NESTED artifacts
vocab_sources.py        *_flat.{json,csv}     vocab_manager      *_nested.json

vocab_sources.py 📎 harvests two controlled vocabularies:

Source	How	What comes back
AMCR heslář	OAI-PMH, `api.aiscr.cz/2.2/oai?set=heslo`	Czech–English pairs plus `ident_cely`, `nazev_heslare` (which of the ~50 controlled lists the term belongs to), `popis`, `zkratka`, `razeni`, broader terms and SKOS mappings
TEATER thesaurus	the 12 pinned `import_*.json` files in `ARUP-CAS/aiscr-teater`, or live `teater.aiscr.cz/api/export`	4 134 concepts in 12 branches, trilingual labels, scope notes, and the real broader/narrower hierarchy

vocab_manager.py 📎 then groups the flat terms into the thematic taxonomy defined by taxonomy_config.json 📎. Placement is tried in precedence order — a per-term correction in taxonomy_overrides.json 📎, AMCR list membership (heslar_map), TEATER branch (teater_branch_map, resolved most-specific-first so a depth-2 sub-branch like muzeum can be moved without moving its whole parent branch), the legacy keyword match, a cross-source rescue, an opt-in LLM fallback, then Other — and every placement records the rule that made it in *_placement_audit.csv, so the grouping can be reviewed rather than taken on trust.

Two labels can collide (AMCR and TEATER both use zámek for "lock" and "château"). vocab_sources.to_term_pairs() treats a same-label group as one concept by default — the winning record's id survives, every other one is listed on it as discarded_ids (issue #6, M7) — and only pulls a record into its own bracketed entry ("zámek (sídlo elity)") when taxonomy_overrides.json explicitly flags it as a genuine homonym (M8). Guessing that from a differing English gloss alone would mistake ordinary translation variance for a real split far more often than it would catch one.

Every vocabulary decision is a config edit, not a code change. The two JSON files are the whole surface a domain reviewer needs; nothing below requires touching Python:

In `taxonomy_config.json` → `_settings`	Decides
`heslar_map`, `teater_branch_map`	which facet a whole AMCR list or TEATER branch lands in, or `__exclude__`
`_exclusions`	why each exclusion stands, and whether it is `settled` or still open (`open_geo_ethnic` / `open_other`)
`geo_guardrail`	whether the prompt's "never select a country/language/region name" clause is in force, and which rules it reaches
`nested_keep`	which harvested keys reach the prompt payload
`admin_stop_words`	what sorts to the back of a facet, and so what survives prompt truncation
`composite_separators`	what splits a composite `X/Y` label
`tie_break`, per-facet `priority`	facet order — load-bearing, since the prompt truncates a prefix

In `taxonomy_overrides.json`, per `(source, id)`	Decides
`facet`	one term's facet, including `"__exclude__"` to drop a single term from a list worth keeping
`sub`	one term's sub-header — otherwise a moved term keeps the header of the list it left
`qualifier_cs`	pull a confirmed homonym out of its dedup group as `"<cs> (<qualifier>)"`
`same_as` / `same_as_suppress`	add or drop a composite/component equivalence link; neither changes what the prompt offers

The system prompt is a config surface too. Its instruction text lives in prompts/system_prompt.txt 📎 as [[named blocks]] in render order; llm_config.txt's PROMPT_* flags choose which of them reach the model, and the run banner prints the resulting on/off list so a log always says what the model was told. PROMPT_GEO_GUARDRAIL is the one three-way switch (strict / preference / off) because the geographic rule has three states, and it is paired with taxonomy_config.json's geo_guardrail.active: prompts/output_template.json 📎 documents the resulting per-document output file.

Reading the prompt needs neither a GPU nor the model stack — prompt_template.py imports nothing outside the standard library, so these run in a bare checkout:

python3 prompt_template.py --blocks                  # which rules are on, and what each costs
python3 prompt_template.py --preview                 # the instruction text, term list elided
python3 prompt_template.py --full  > prompt.txt      # the whole prompt, all 4 718 terms
python3 prompt_template.py --diff PROMPT_GEO_GUARDRAIL=strict \
                                 PROMPT_GEO_GUARDRAIL=preference
python3 prompt_template.py --write                   # regenerate the committed sheets
python3 prompt_template.py --check                   # exit 1 if a sheet is out of date

--full renders the untruncated prompt — instructions under the current flags, then every term VOCAB_PATH offers, grouped exactly as build_system_prompt() groups them (both call the same two functions, and a test asserts they agree byte for byte). It is what a model with room for the whole vocabulary sees; at a tighter window the run drops a tail of terms, which context_budget.csv 📎 sizes per window.

The four sheets under prompts/ 📎 — prompt_blocks.txt, prompt_preview.txt, prompt_full.txt, prompt_guardrail_diff.txt — are the output of the first four commands, committed so a reviewer can read the prompt in a diff without running Python. They are generated, never hand-edited: --write rewrites them, --check fails when the vocabulary, a flag or the template has moved without them, and .github/workflows/vocab-drift.yml runs that check on every PR touching either half.

PROMPT_VOCAB_GROUPING controls the layout of the term list, not its contents: facet_sub (shipped — --- Facet / Subgroup ---, both curated levels), facet (facet headers only) or flat (no headers). It exists to answer @motyc's question in issue #6 — whether the facet grouping affects results at all, now that the whole vocabulary fits a 128k window. All three offer the same terms and truncate identically; facet and flat also preserve term order, while facet_sub makes each facet's sub-groups contiguous. So facet vs flat isolates the ~125 header lines and facet_sub vs facet measures the source's second level. The headers are not free: at an 8 192-token window they cost 26 terms, at 32 768 they cost 126, and at 128k nothing, since everything fits either way.

validate_settings() refuses an edit that would not do what it says — an undeclared facet, a relabel for a list no map places, a reason for something nobody excludes, an unknown override key, a stale (source, id), a pair both linked and suppressed, two overrides that would build the same bracketed key — and reports every problem at once rather than one per rebuild. vocab_build.py additionally renders the prompt the config selects and refuses to build a vocabulary that contradicts its geographic guardrail. Two runbooks carry the operational detail, and they are the pages to read before touching either half: data_samples/vocab/RUNBOOK.md 📎 — every vocabulary script, the eight review sheets, and the full "where a decision gets recorded" table; prompts/RUNBOOK.md 📎 — the eleven prompt blocks, the nine flags, the guardrail's two halves, and the output contract.

# stage 1 + 2, needs network access to aiscr.cz
python3 vocab_build.py --source both --stats

# stage 2 only: re-nest from the committed flat files after editing the taxonomy.
# Pure, offline, sub-second — this is the loop for tuning the taxonomy.
python3 vocab_build.py --from-flat --stats
python3 vocab_build.py --from-flat --check     # exit 1 if the artifacts would change

If this machine cannot reach aiscr.cz, run the Vocabulary Refresh workflow (.github/workflows/vocab-refresh.yml) — a hosted runner harvests and uploads the artifacts.

[!NOTE] The nested files are deliberately not written with sort_keys=True. Theme order is priority-descending and load-bearing: build_system_prompt() iterates the file in insertion order and truncates a prefix of the resulting term list, so alphabetising the keys would silently change which themes survive a tight context budget. Determinism comes from the explicit priority ordering plus a (boilerplate, razeni, label) sort within each theme. Provenance lives in a sidecar *.meta.json, not inline — every consumer reads the nested file as {theme: terms}, so an inline _meta key would be rendered to the model as a phantom theme.

python3 vocab_manager.py still works and still performs the legacy AMCR-only sync.

Reviewing the vocabulary

Two read-only tools turn the built vocabulary into sheets a domain reviewer can rule on. Neither writes to the vocabulary, and neither guesses a semantic verdict — they rank and surface candidates; a human decides, and the decision goes back as a config edit.

vocab_review.py 📎 — eight sheets, offline and pure, built from the committed *_flat.json plus the taxonomy config:

python3 vocab_review.py --all            # all eight, into data_samples/vocab/
python3 vocab_review.py --collisions     # collision_review.csv        same-label groups (M8/M13)
python3 vocab_review.py --composites     # composite_pairs.csv         "X/Y" vs standalone X, Y
python3 vocab_review.py --exclusions     # exclusion_impact.csv        what each exclusion costs
python3 vocab_review.py --subbranches    # teater_subbranch_impact.csv the same, one level finer
python3 vocab_review.py --reinstate      # reinstatement_preview.csv   usable count + token delta
python3 vocab_review.py --specificity    # specificity_pairs.csv       term offered under its own parent
python3 vocab_review.py --budget         # context_budget.csv          what survives each window
python3 vocab_review.py --census         # facet_census.csv            what is in each facet, and its cost

corpus_review.py 📎 — three evidence sheets matching the vocabulary against real report text by UDPipe lemma, which is the evidence standard for keeping or dropping a branch:

python3 corpus_review.py --all           # corpus_term_evidence.csv, corpus_branch_evidence.csv,
                                         # gold_workbook.csv (+ corpus_review.meta.json)

⚠️ corpus_review.py needs the document corpus, and the repository tracks only three synthetic demo documents — the real reports arrive as the issue #19 attachment and are untracked. Every run prints its corpus size before writing, and the tool refuses to overwrite sheets built from a larger corpus rather than silently replacing real evidence with placeholder numbers. On a clean checkout these three are a smoke test, not evidence.

Both sets are covered by a drift test: a taxonomy edit that is not followed by a regeneration fails tests/test_vocab_review.py, for the same reason the artifacts have their own gate. See data_samples/vocab/RUNBOOK.md 📎 for what each sheet answers and how to read it.

2. LLM Inference Pipeline (llm_run.py 📎)

Reads the CSV files, filters lines by quality, injects the nested vocabulary and a sliding context window into the system prompt, and executes constrained generation. Output files are named <stem>_enriched.json and written to KW_PER_DOC_LLM_<model_suffix>/.

[!TIP] All model-loading logic, constrained-decoding helpers, and prompt templates live in llm_utils.py 📎 and are shared between both backends.

# Transformers backend (default)
python3 llm_run.py

# Custom config file
python3 llm_run.py my_config.txt

For multi-GPU runs (vLLM backend):

# 1. Edit llm_config.txt:
#    BACKEND=vllm
#    MODEL_KEY=qwen3-235b-a22b-fp8
#    TENSOR_PARALLEL_SIZE=2
#    ENABLE_PREFIX_CACHING=true
python3 llm_run.py

🖥 Model Registry

The built-in registry in llm_utils.py covers the full range of supported models. All VRAM figures assume BnB 4-bit for the transformers backend and FP8/BF16 for vLLM.

Single-GPU — `BACKEND=transformers` (or `BACKEND=vllm`)

Registry key	Model	Size	Context	Est. VRAM	Notes
`qwen-3.6-27b-it`	Qwen/Qwen3.6-27B ⁶	27 B dense	262 k	~18 GB	Default. Best accuracy/VRAM ratio on a single GPU.
`gemma-4-31b-it`	google/gemma-4-31B-it ⁷	31 B dense	256 k	~21 GB	Highest single-GPU accuracy. Gated — `HF_TOKEN` required.
`qwen3-14b`	OpenPipe/Qwen3-14B-Instruct ⁸	14 B dense	128 k	~9 GB	Good baseline; thinking mode suppressed automatically.
`qwen-3.5-9b-it`	Qwen/Qwen3.5-9B ⁹	9 B dense	262 k	~6 GB	Entry-level (8 GB VRAM).
`qwen3-8b`	Qwen/Qwen3-8B ¹⁰	8 B dense	128 k	~16 GB	BF16 (no 4-bit); straightforward baseline.
`qwen2.5-14b-awq`	Qwen/Qwen2.5-14B-Instruct-AWQ ¹¹	14 B AWQ	128 k	~9 GB	Pre-quantized; fast on NVIDIA GPUs.
`qwen2.5-7b`	Qwen/Qwen2.5-7B-Instruct ¹²	7 B dense	32 k	~14 GB	BF16; short context window.
`gemma-3-12b-it`	google/gemma-3-12b-it ¹³	12 B dense	128 k	~8 GB	Good bilingual extraction. Gated.

MoE models — GGUF / llama.cpp fallback (any GPU, any VRAM)

Registry key	Model	Active params	Context	Notes
`gemma-4-26b-moe-gguf`	bartowski/google_gemma-4-26B-A4B-it-GGUF	4 B	8 k	BnB 4-bit unsupported (fused experts). Q4_K_M quantization via llama.cpp.

MoE models — `BACKEND=vllm` (single GPU or multi-GPU)

Registry key	Model	Active params	Context	Notes
`qwen-3.6-35b-moe`	Qwen/Qwen3.6-35B-A3B ¹⁴	3 B	262 k	35 B total / 3 B active. Single GPU usually fits.
`gemma-4-26b-moe`	google/gemma-4-26B-A4B-it ¹⁵	4 B	256 k	26 B total / 4 B active. Gated.
`gemma-4-26b-moe-awq`	google/gemma-4-26B-A4B-it ¹⁵	4 B	256 k	AWQ-quantised variant of `gemma-4-26b-moe`. Gated.

Large models — `BACKEND=vllm`, `TENSOR_PARALLEL_SIZE ≥ 2`

Registry key	Model	Total / Active	Context	Rec. TP	Notes
`qwen3-235b-a22b-fp8`	Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 ¹⁶	235 B / 22 B	128 k	2	Recommended for 144 GB / 200 GB nodes. Native FP8 (~117 GB loaded).
`qwen3-235b-a22b`	Qwen/Qwen3-235B-A22B-Instruct-2507 ¹⁶	235 B / 22 B	128 k	2	BF16 variant — heavier than FP8.
`deepseek-v3`	deepseek-ai/DeepSeek-V3 ¹⁷	671 B MoE / —	128 k	4	FP8 official checkpoint available. 4×80 GB minimum.
`llama4-maverick`	meta-llama/Llama-4-Maverick-17B-128E-Instruct ¹⁸	128 experts / 17 B active	1 M	2	Multimodal. 1 M token context. Gated — `HF_TOKEN` required.
`llama3.1-70b`	meta-llama/Meta-Llama-3.1-70B-Instruct ¹⁹	70 B dense / —	128 k	2	Also works with `transformers` + 4-bit on 2×40 GB. Gated.

[!TIP] Automatic Prefix Caching (APC) — enabled by default for the vLLM backend (ENABLE_PREFIX_CACHING=true). The system prompt (which embeds the full TEATER vocabulary) is computed once per run; its KV-cache is reused across every line in every document. This is the primary throughput multiplier: on a 500-line document the vocabulary forward pass happens once instead of 500 times. APC also removes the need to truncate the vocabulary to fit the token budget — the full thematic dictionary is injected when APC is active.

📁 Inputs and Outputs

Input: DOC_LINE_CATEG/*.csv (contains file_id, page_num, line_num, categ, quality_score, and raw text).
Output: KW_PER_DOC_LLM_<model_suffix>/*_enriched.json — one file per document, containing an array of JSON objects that merge CSV metadata with the LLM's semantic extraction.
Abort sidecar: KW_PER_DOC_LLM_<model_suffix>/*_enriched.abort.json — written alongside the main output only when a document is abandoned after 10 consecutive inference errors. Its presence is the canonical signal that the corresponding JSON file contains partial results.

Example output record:

{
  "file_id": "CTX195603828",
  "page": 1,
  "line": 14,
  "categ": "Text",
  "quality_score": 0.98,
  "original_text": "Výzkum odhalil základy gotického kostela ze 14. století.",
  "enrichment": {
    "extracted_keywords_cs": ["základy", "gotický kostel"],
    "extracted_keywords_en": ["foundations", "gothic church"],
    "teater_category": "kostel",
    "teater_category_ids": [
      { "source": "amcr", "id": "HES-000021" },
      { "source": "amcr", "id": "HES-000465" },
      { "source": "teater", "id": "1333" }
    ],
    "confidence_score": 0.95
  }
}

teater_category_ids (present when EMIT_CATEGORY_IDS=true, the default) lists every source record the selected vocabulary term absorbed during dedup — issue #6, M7. It is attached after inference, from the vocabulary's own discarded_ids; the prompt and schema are unaffected. When the selected term was a bracketed disambiguation (B3, e.g. "zámek (sídlo elity)"), teater_category is stripped back to the bare label ("zámek") before it is written, and teater_category_ids carries the id that disambiguates which sense was meant — a term that legitimately contains parentheses in its own source label (e.g. "GPS (navigační systém)") is never touched.

Abort sidecar format (*_enriched.abort.json):

{
  "aborted": true,
  "abort_reason": "10 consecutive inference errors",
  "processed_before_abort": 42,
  "errors_before_abort": 10,
  "timestamp_utc": "2026-05-20T09:14:33"
}

[!NOTE] None of the per-model output sets below is committed to this repository — each is the real output of a real run against the full report corpus, not a sample bundled with the code (same reason data_samples/DOC_LINE_CATEG/ itself holds only three synthetic demo documents). The directory names are the OUTPUT_DIR a local run with that MODEL_KEY produces; the footnote on each is the model card.

Output examples per model (directory names, not links — see the note above):

KW_PER_DOC_LLM_qwen3_14b by Qwen 3-14B ⁸
KW_PER_DOC_LLM_qwen25_14b_awq by Qwen 2.5-14B AWQ ¹¹
KW_PER_DOC_LLM_gemma_3_12b_it by Gemma 3-12B-IT ¹³
KW_PER_DOC_LLM_qwen_36_27b_it by Qwen 3.6-27B-IT ⁶
KW_PER_DOC_LLM_gemma_4_31b_it by Gemma 4-31B-IT ⁷
KW_PER_DOC_LLM_qwen_35_9b_it by Qwen 3.5-9B-IT ⁹
KW_PER_DOC_LLM_llama31_70b by LLaMA 3.1-70B ¹⁹
KW_PER_DOC_LLM_qwen3_8b by Qwen 3-8B ¹⁰

Pending (sample runs in progress):

KW_PER_DOC_LLM_qwen_36_35b_moe by Qwen 3.6-35B-MoE ¹⁴
KW_PER_DOC_LLM_gemma_4_26b_a4b_it by Gemma 4-26B-A4B-IT ¹⁵

Archived (unsuccessful — evaluation notes in issue #6; would have been under archived_KW_PER_DOC_LLM/):

KW_PER_DOC_LLM_mistral_nemo_12b by Mistral Nemo 12B ²⁰
KW_PER_DOC_LLM_aya_expanse_8b by Aya Expanse 8B ²¹
KW_PER_DOC_LLM_bielik_11b_v30 by Bielik 11B v3.0 ²²
KW_PER_DOC_LLM_llama31_8b by LLaMA 3.1-8B ²³
KW_PER_DOC_LLM_ministral_3_14b by Ministral 3-14B ²⁴
KW_PER_DOC_LLM_qwen3_8b (early run) by Qwen 3-8B ¹⁰
KW_PER_DOC_LLM_qwen25_7b by Qwen 2.5-7B ¹²

📊 Paradata Integration

Just like the main shell-script pipelines, LLM enrichment natively hooks into atrium_paradata.py and automatically logs:

Full snapshot of llm_config.txt 📎 and quality-filter settings.
Which vocabulary build was used — read from the *.meta.json beside the artifact: tool version, term count, the sha256 of both taxonomy files, and each source's record count and pinned ref (TEATER's harvest commit). A run is reproducible only if the vocabulary it saw is identifiable, and every placement decision is a function of those two sha256s.
Both vocabulary sources as licence components. The AMCR heslář and the TEATER thesaurus are CC BY-NC 4.0 and declared conditional in para_config.txt 📎, so they constrain a run's effective licence only when log_component() names them. Logged per source actually present in the build — an AMCR-only artifact does not claim it used TEATER data.
Total processed lines (json success events).
Per-line tracking of filter skips (skipped_filter), inference faults (skipped_error), and already-completed files (already_exists).
Abort events — when a document is abandoned after 10 consecutive inference errors, the paradata entry records the abort reason alongside the count of lines processed before the failure. A sidecar *.abort.json file is also written next to the (partial) output JSON for easy programmatic detection. The resulting logs are dropped into the specified PARADATA_DIR alongside the other pipeline execution records.

EXTRA: REST API Service

The pipeline now includes a fully-featured FastAPI REST service that exposes the core NLP enrichment and rescaling functionalities over HTTP.

Single-file enrichment: Upload CSV, XLSX, or plain text to the /enrich endpoint and receive a combined JSON envelope (or ZIP workspace) with TEITOK XML, keywords, paradata, and NER summaries.
Coordinate Rescaling: Use the /rescale endpoint to align XML spatial coordinates to specific target image resolutions directly over the network.
Job Management: Background processing for larger documents with a asynchronous /jobs queue.

For complete setup instructions, payload examples, and endpoint documentation, refer to the Service README.

Paradata Logs

Every pipeline script records structured provenance metadata through atrium_paradata.py 📎. Two complementary log surfaces are produced after a run:

`<OUTPUT_DIR>/paradata/` — structured run logs 📂

Each of the four pipeline scripts produces one JSON file here, named with the pattern:

YYMMDD-HHmmss_nlp-enrich.json

where the timestamp prefix is the UTC wall-clock time at which the script started. Because every script is an independent invocation, a complete four-step run will create four separate files, making it straightforward to audit individual stages in isolation.

The paradata logs (samples in directory paradata 📂) capture key details about each pipeline stage, including the program name, run ID, execution duration, configuration parameters, input and output statistics, and performance metrics. They also document skipped files with reasons and provide a breakdown of output types and processing rates for benchmarking. This structured metadata ensures traceability and facilitates auditing of the pipeline's execution.

The declared output types per stage are:

Script	Types recorded
`api_1_manifest.sh`	`tsv` (one entry per input CSV/XLSX processed into the manifest)
`api_2_udp.sh`	`conllu` (one per document)
`api_3_nt.sh`	`tsv` (one per page — count reflects individual page TSV files)
`api_4_stats.sh`	`csv` always; `conllu` when `SAVE_CONLLU_NE=true`; `xml` when `SAVE_TEITOK=true`
`keywords.py`	`csv_per_doc` (one per document keyword CSV) and `csv_summary_row` (one summary row per document)

[!NOTE] When resuming an interrupted run (steps 2–4 skip already-finished documents via [ -f "$out" ] && continue), the resumed documents are not re-counted in the paradata JSON. The input_files_total field still reflects the full manifest, so skipped_files + successfully_processed will be less than input_files_total for partial runs. This is expected behaviour; the difference represents the documents carried over from a previous invocation.

[!NOTE] Paradata state files. While a pipeline script is running, atrium_paradata.py stores intermediate state in a plain-text JSON file inside <OUTPUT_DIR>/paradata/ (named .state_<runid>_<program>.json). This file is automatically removed when the script completes. Because it is plain JSON it can be inspected with any text editor if a run is interrupted unexpectedly.

`<OUTPUT_DIR>/processing.log` — human-readable runtime log 📄

api_common.sh 📎 exposes a log() helper that timestamps and tee-appends warnings and errors to this flat file. The four main pipeline scripts (api_1_manifest.sh … api_4_stats.sh) write to processing.log indirectly through the Python helpers, which print timestamped messages to stderr; any script that sources api_common.sh can also write here via the log() function directly.

[2026-01-15 09:42:11] [WARN] UDPipe failed (HTTP 503). Retrying in 2s…
[2026-01-15 09:42:14] [ERR]  UDPipe failed permanently after 5 attempts.

This file accumulates across reruns; it is the first place to check when a document appears in skipped_files_detail but the reason is terse.

`TEMP/` — intermediate working files 📂

TEMP/ (set by WORK_DIR in config_api.txt 📎) holds transient artefacts that are only needed during processing and can be deleted once the full pipeline has completed successfully:

TEMP/
├── CHUNKS/
│   ├── <doc_id>/
│   │   ├── chunk_0.txt      # OCR-line-preserving text fragment sent to UDPipe
│   │   ├── chunk_1.txt
│   │   └── …
│   └── …
└── nametag_response_<doc_id>.conllu.json   # raw JSON reply from the NameTag API

CHUNKS/ is produced by api_util/chunk.py 📎 which splits documents that exceed WORD_CHUNK_LIMIT (default 900 words) into sentence-boundary-aware fragments before each UDPipe API call. Each chunk file preserves the original OCR line structure (one line per row) so that UDPipe receives correct sentence-boundary signals between text lines. The per-chunk plain-text files and the raw NameTag JSON responses carry no provenance value after the CoNLL-U files have been merged and validated; they are not tracked by the paradata logger.

[!TIP] If disk space is a concern you can safely delete TEMP/ once <OUTPUT_DIR>/UDP/ and <OUTPUT_DIR>/NE/ have been fully populated and step 4 has completed without errors. The paradata JSONs in <OUTPUT_DIR>/paradata/ and the processing.log are the only runtime records worth keeping long-term.

One-command pipeline run (`run_pipeline.py`)

While each stage can be launched manually (see Workflow Stages), run_pipeline.py 📎 chains them end-to-end and merges every per-stage paradata JSON produced during the run into a single pipeline-run-merged record.

# Full core run: api_1 → api_2 → api_3 → api_4
python3 run_pipeline.py

# Core run plus keyword extraction (CPU-only YAKE backend, default)
python3 run_pipeline.py --kw

# Keyword extraction with the GPU KeyBERT backend
python3 run_pipeline.py --kw --kw-method keybert

# Add the optional LLM semantic-enrichment stage (needs requirements_llm.txt)
python3 run_pipeline.py --kw --llm

# Run only a subset of the core stages (canonical order is always enforced)
python3 run_pipeline.py --stages udp nt

# Resume after an interruption: start from a chosen stage, skip every earlier one
python3 run_pipeline.py --start-from nt

# Skip individual stages (re-run only NER + stats, leave manifest/UDPipe as-is)
python3 run_pipeline.py --skip-manifest --skip-udp

# Clear stale .state_* checkpoint sidecars from PARADATA_DIR before running
python3 run_pipeline.py --clean-state

# Force execution: bypass missing dependency checks and ignore individual stage failures
python3 run_pipeline.py --kw --kw-method keybert --force

# Validate configuration and resolve the plan without running anything
python3 run_pipeline.py --dry-run

# Print the resolved config + stage plan as JSON (for wrappers / healthchecks)
python3 run_pipeline.py --print-config json

The runner reads the same config_api.txt 📎 that the shell stages source, so Python and Bash always agree on OUTPUT_DIR, PARADATA_DIR, and the input/output paths.

What the runner does

Resolves config from config_api.txt 📎 (with $VAR / ${VAR} expansion).
Runs each stage in order, spacing stage starts by ≥ 1.1 s so the 1-second-resolution paradata filenames (YYMMDD-HHmmss_nlp-enrich.json) never collide.
Collects the paradata JSON each stage writes, scoped to this run only (paradata files that already existed before the run are never merged).
Merges all per-stage records into one <PARADATA_DIR>/<runid>_nlp-enrich_pipeline-run.json via atrium_paradata.merge_run_paradata. The merged record accurately tracks document-level statistics across the sequential pipeline (recording true throughput without inflating input counts). The effective license of the merged record is re-derived from the union of every component used across the stages, so the most-restrictive rule holds end-to-end (a core run is CC BY-NC-SA 4.0; adding the YAKE backend escalates the share-alike/AGPL constraint, etc.).

Resume / checkpoint recovery

Long batches on constrained hardware are expensive to restart from scratch, so the runner lets you re-enter the pipeline at any stage instead of redoing completed work. Recovery operates at two complementary levels.

Document-level (automatic). Every stage already skips inputs whose output exists (steps 2–4 via [ -f "$out" ] && continue; the LLM stage logs already_exists and moves on), so simply re-running the same command picks up where the previous run stopped. When the LLM stage abandons a document after 10 consecutive inference errors it writes a *_enriched.abort.json sidecar next to the partial output (see LLM Inputs and Outputs); that marker is the canonical signal that a document holds partial results and should be re-run.

Pipeline-level starting points. To skip whole stages — not just completed documents — the runner accepts explicit entry points over the full stage order:

Flag	Effect
`--start-from <stage>`	Run from `<stage>` onward; every earlier stage is skipped.
`--skip-<stage>`	Skip one named stage, run the rest.
`--clean-state`	Sweep stale `.state_*.json` sidecars from `PARADATA_DIR` before running.

<stage> is one of manifest, udp, nt, stats, keywords, llm (the canonical order; keywords/llm require their --kw/--llm flags to be part of the run). Each skip flag also has an equivalent SKIP_<STAGE>=true knob that can live in config_api.txt 📎 (e.g. SKIP_MANIFEST=true), so a habitual resume profile can be persisted without retyping flags.

# UDPipe + NameTag already finished — resume at statistics, then keywords
python3 run_pipeline.py --kw --start-from stats

# Re-run only NER and statistics; keep the existing manifest and CoNLL-U
python3 run_pipeline.py --skip-manifest --skip-udp

Skipped stages are recorded under skipped_stages in the merged <runid>_nlp-enrich_pipeline-run.json record, so a resumed run remains fully auditable. An all-skipped run is treated as a successful resume, not an empty failure (see Exit codes below).

Provenance for containers

When the runner (or its Docker entrypoint) is started with the ATRIUM_RUNNER_IMAGE, ATRIUM_RUNNER_REPO, and ATRIUM_RUNNER_REF environment variables set, those values are forwarded to every stage subprocess and end up in each stage's paradata record (and therefore the merged record). This ties a run back to the exact image/commit that produced it.

ATRIUM_RUNNER_IMAGE="ghcr.io/ufal/atrium-nlp-enrich:v0.11.0" \
ATRIUM_RUNNER_REF="$(git rev-parse --short HEAD)" \
python3 run_pipeline.py --kw

Exit codes

Code	Meaning
`0`	All requested stages completed; nothing flagged.
`1`	A stage processed nothing despite having input and no resume, and `FAIL_ON_EMPTY=true` (the default).
`2`	A required stage script was not found.
`3`	A dependency preflight failed (e.g. `--kw-method keybert` without `keybert`/`sentence-transformers`, or `--llm` without the requirements_llm.txt 📎 stack).
`≠0`	A stage script itself exited non-zero (its code is propagated).

[!TIP] Using --force (-f) overrides exit codes 1, 3, and ≠0. It bypasses preflight dependency crashes and forces FAIL_ON_EMPTY=False, allowing the pipeline to continue attempting subsequent stages even if one stage crashes or processes zero files.

The empty-run guard is governed by FAIL_ON_EMPTY in config_api.txt 📎. A resumed run — where every document was already complete and thus skipped, or where a stage was skipped outright via --start-from / --skip-<stage> (see Resume / checkpoint recovery) — is treated as success, not an empty failure. Set FAIL_ON_EMPTY=false to permit genuinely empty stages.

[!NOTE] The runner never re-implements stage logic: it shells out to the exact same api_1_manifest.sh … api_4_stats.sh, config_api.txt, and llm_run.py 📎 you can run by hand. Anything documented for those stages (resume behaviour, output flags, model registry, …) applies unchanged under the runner.

Acknowledgements 🙏

For support write to: lutsai.k@gmail.com responsible for this GitHub repository ²⁵ 🔗

Developed by UFAL ²⁶ 👥
Funded by ATRIUM ²⁷ 💰
Shared by ATRIUM ²⁷ & UFAL ²⁶ 🔗
Frameworks used:
- Lindat/CLARIAH-CZ NameTag 3 API ³ 🏷
- Lindat/CLARIAH-CZ UDPipe 2 API ² 🏷
- local KER (original lemma-frequency keyword extraction) ²⁸ 🏷
- YAKE (Yet Another Keyword Extractor, CPU statistical keyword extraction) ²⁹ 🏷
- KeyBERT (embedding-based keyword extraction, GPU-accelerated) ³⁰ 🏷
- UFAL flexiconv (format conversion to TEITOK XML) ⁵ 🏷

©️ 2026 UFAL & ATRIUM

Contributors

K4TEL

364 commits

dependabot[bot]

17 commits

Similar repos

ufal/atrium-llm-enrich

Application of LLMs in context of the ATRIUM project for digitized documents enrichment

same org

UCREL/mosaico-usas-processing

Processing of the Mosaico dataset with USAS and CLAWS

impresso/impresso-named-entity-data-augmentation

Production of synthetic data to augment named entity ground truth

Digital-Scholarship-Oxford/crowdsourced-data-tools

Crowdsourced data tools

stonesalltheway1/Epstein-Pipeline

Open source document processing pipeline for the Epstein case files. Download OCR, extract…

116

adbar/trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction,…

6,798

michaelscho/lassberg

diazmarteth/NER

Named Entity Recognition pipeline with AI integration for Art and Architecture History.

Languages

Python

97.5%

Shell

1.5%

ufal/atrium-nlp-enrich

NLP enrichment of text lines from ALTO XML files

stars

381

commits

Python

primary language

Sep 8, 2026

updated

named-entity-recognition

part-of-speech-tagging

README

📦 ALTO XML Files Postprocessing Pipeline - NLP Enrichment of text

[!CAUTION] This repository is a follow-up to main ALTO XML postprocessing GitHub repository, a part of ATRIUM project dedicated to ALTO-2-TXT workflow and collection of statistics and from text content of the documents (text and bounding boxes ordered by LayoutReader) recorder in CSV (XLSX) tables as a text column ¹.

TEITOK XML — Unified Output Format
⚙️ Setup
Workflow Stages
- Step 1: Prepare CSVs with texts from Page-Specific ALTOs
- Step 2: Extract NER and CONLL-U
  - Configuration ⚙️
  - Execution Pipeline
Output Structure
EXTRA: Extract Keywords (KER / YAKE / KeyBERT)
EXTRA: Converting Other Input Formats with flexiconv
EXTRA: LLM Semantic Enrichment (Vocabulary Mapping)
- Reviewing the vocabulary · runbooks: vocabulary · prompt & output
EXTRA: REST API Service
Paradata Logs
Acknowledgements

TEITOK XML — Unified Output Format

Each document in the collection is serialised as a single .teitok.xml file that integrates four layers of information in a consistent, machine-readable structure:

Layer	Content
Layout	Page, text-block, and line boundaries with pixel-accurate bounding boxes from the source ALTO XML, scaled to match the stored PNG images
Morphology & Syntax	Per-token lemma, UPOS/XPOS tags, morphological features, and dependency relations produced by UDPipe 2
Named Entities	BIO-tagged entity spans with both a CoNLL-style category (`PER`, `ORG`, `LOC`, `MISC`) and a fine-grained CNEC 2.0 code (e.g. `pf` = first name, `gu` = city) produced by NameTag 3
Facsimile links	`<surface>` elements in `<facsimile>` that tie each page to its companion image, enabling TEITOK's side-by-side text/image view

Why TEITOK XML?

Storing all enrichment layers in a single interoperable format offers several practical advantages over keeping CoNLL-U, TSV, and image files in separate silos:

🔍 Full-text and attribute search — TEITOK's built-in CQL/XPATH query engine lets users search across lemmas, NER types, POS tags, and raw text simultaneously.
🏷 Named entity access — entity spans (<name type="PER" cnec="pf">) are first-class XML elements: queryable, stylable, and exportable independently of the surrounding tokens.
🖱 Mouseover information — hovering over any token in the TEITOK GUI surfaces its lemma, morphological features, and dependency relation without leaving the page view.
🖼 Page visualisation with spatial overlays — bounding box coordinates on every <tok>, <lb>, and <div> are used by TEITOK's facsimile viewer to overlay text highlights directly onto the scanned page image, making OCR quality immediately visible.
📐 Layout-aware structure — text blocks (<div type="MarginTextZone-P">), lines (<lb>), and graphical elements (<figure>) preserve the physical layout of the original document.
🔗 Interoperability — TEI/XML is a widely adopted standard in digital humanities; the files can be ingested by other TEI-aware tools (e.g. eXist-db, Oxygen XML Editor) without conversion.

TEITOK XML structure at a glance

<TEI xmlns="http://www.tei-c.org/ns/1.0" xml:lang="cs">
  <teiHeader> ... </teiHeader>

  <facsimile>
    <surface id="doc1.surface1" lrx="1240" lry="1754">
      <graphic url="doc1-1.png"/>
    </surface>
  </facsimile>

  <text><body>
    <pb n="1" id="doc1.pb1" facs="doc1-1.png"/>

    <div type="MarginTextZone-P" id="doc1.TB_1" bbox="142 210 1098 880">
      <s id="doc1.s1" text="Výroční zpráva 2012 .">
        <lb id="doc1.TL_1" bbox="142 210 680 255"/>

        <tok id="doc1.s1.w1" type="w" lemma="výroční" upos="ADJ"
             feats="Case=Nom|..." deprel="amod"
             bbox="142 210 310 255">Výroční</tok>

        <name type="ORG" cnec="if">
          <tok id="doc1.s1.w3" type="w" lemma="ministerstvo" upos="NOUN"
               bbox="320 210 580 255">Ministerstvo</tok>
          <tok id="doc1.s1.w4" type="w" lemma="finance" upos="NOUN"
               bbox="585 210 680 255">financí</tok>
        </n>
      </s>
    </div>
  </body></text>
</TEI>

[!NOTE] TEITOK XML is generated by Step 4 of this pipeline (api_4_stats.sh) when SAVE_TEITOK=true. The source ALTO XML files must be present in INPUT_ALTO_DIR for spatial coordinates to be included. If INPUT_ALTO_DIR is not set, TEITOK XML is still produced but without bounding box attributes. If your documents are not in ALTO format, see EXTRA: Converting Other Input Formats with flexiconv.

⚙️ Setup

Before you begin, set up your environment.

Create and activate a new virtual environment in the project directory 🖥.
Install the required Python packages:

pip install -r requirements.txt

For keyword extraction, install the backend(s) you intend to use:

# YAKE — unsupervised statistical extraction, CPU-only
pip install yake

# KeyBERT — embedding-based extraction, GPU-accelerated when available
pip install keybert sentence-transformers
pip install torch          # optional — enables CUDA GPU acceleration

The original legacy KER backend requires no additional packages. For the LLM Semantic Enrichment pipeline, install the inference backend you intend to use:

# Transformers backend — single GPU, models ≤ 31 B (BnB 4-bit / AWQ / GGUF)
pip install -r requirements_llm.txt

# vLLM backend — multi-GPU, large models (≥ 70 B), Automatic Prefix Caching
# Replaces lmformatenforcer; uses xgrammar for native guided JSON decoding
pip install vllm

(Optional) To run the REST API service, install additional requirements:

pip install -r service/requirements.txt

Review and update the config_api.txt 📎 file with your specific paths and API configurations. You are now ready to start the workflow.

Workflow Stages

The process is divided into sequential steps, each responsible for a specific part of the NLP enrichment pipeline.

▶ Step 1: Prepare CSVs with texts from Page-Specific ALTOs

[!IMPORTANT] If you already have a directory of CSV (XLSX) tables with text column containing extracted text files from ALTO XMLs, you can skip Step 1 and proceed directly to Step 2.

CSVS_with_TEXT/
├── document1.csv
├── document2.csv
└── ...

with the structure of each CSV (XLSX) file like:

file,page_num,line_num,text,split_ws,split_we,lang,lang_score,perplex,categ
CTX201504033,1,8,2012,,,N/A,0,0,Non-text
CTX201504033,2,2,1,,,N/A,0,0,Non-text
CTX201504033,3,2,2,,,N/A,0,0,Non-text
...

If the script detects an .xlsx file, it will iterate over all sheet names, verify if a text column exists in each sheet, and extract the content safely for Excel tables with multiple sheets.

▶ Step 2: Extract NER and CONLL-U

This stage performs advanced NLP analysis using external APIs (Lindat/CLARIAH-CZ) to generate Universal Dependencies (CoNLL-U) and Named Entity Recognition (NER) data.

Unlike previous steps, this process is split into modular shell scripts to handle large-scale processing, text chunking, and API rate limiting.

Configuration ⚙️

Before running the pipeline, review the api_config.txt 📎 file. This file controls directory paths, API endpoints, and model selection.

# config_api.txt
OUTPUT_DIR="../../ARUB"                          # Destination for results
INPUT_TABLES_DIR="$OUTPUT_DIR/DOC_LINE_LR_CLS"  # Input tables from Step 1

WORK_DIR="./TEMP"                                # Working directory for intermediate files

LOG_FILE="$OUTPUT_DIR/processing.log"
CONLLU_INPUT_DIR="$OUTPUT_DIR/UDP"
TEMP_TXT_DIR="./TEMP/TXT_EXTRACT"
CHUNK_DIR="./TEMP/CHUNKS"

TSV_INPUT_DIR="$OUTPUT_DIR/NE"
SUMMARY_OUTPUT_DIR="$OUTPUT_DIR/UDP_NE"

TEITOK_OUTPUT_DIR="$OUTPUT_DIR/TEITOK"
INPUT_ALTO_DIR="$OUTPUT_DIR/altos"              # Source ALTO XML files - for TEITOK conversion
# ── Image Options ─────────────────────────────────────────────────────────────
# OPTIONAL: Only required if your companion PNG/JPEG display images have been resized
# to a different target resolution relative to ABBYY's baseline dimensions.
# If left empty, the pipeline calibrates layout shifts natively using ALTO PrintSpace.
INPUT_PAGES_DIR=""

UDPIPE_URL="https://lindat.mff.cuni.cz/services/udpipe/api/process"
NAMETAG_URL="https://lindat.mff.cuni.cz/services/nametag/api/recognize"

MODEL_UDPIPE="czech-pdt-ud-2.15-241121"
MODEL_NAMETAG="nametag3-czech-cnec2.0-240830"

TIMEOUT=60                     # API call timeout in seconds
MAX_RETRIES=5                  # Number of retries for failed API calls
BACKOFF_FACTOR=1.5
WORD_CHUNK_LIMIT=900           # Word limit per API call

SAVE_CSV=true                  # write token-level summary CSV
SAVE_CONLLU_NE=true            # keep merged CoNLL-U with NER in MISC
SAVE_TEITOK=true               # write TEITOK-style TEI XML (flexiconv-compatible)

Execution Pipeline

1. Generate Manifest

Maps input text files to document IDs and page numbers to ensure correct processing order.

./api_1_manifest.sh

Input: ../CSVS_with_TEXT/ (raw text files in subdirectories from Step 1).
Output: OUTPUT_DIR/manifest.tsv.

tail -n +2 OUTPUT_DIR/manifest.tsv | wc -l

which returns the total number of document rows in the manifest, excluding the header line.

2. UDPipe Processing (Morphology & Syntax)

Sends text to the UDPipe API ². Large documents are automatically split into chunks (default 900 words) using chunk.py 📎 to respect API limits, then merged back into valid CoNLL-U files.

./api_2_udp.sh

Input 1: OUTPUT_DIR/manifest.tsv (mapping of text files to document IDs and page numbers).
Input 2: ../CSVS_with_TEXT/ (raw text files in subdirectories from Step 1).
Output: OUTPUT_DIR/UDP/*.conllu (Intermediate per-document CoNLL-U files).

Run the following command to see how many documents have been processed into CoNLL-U files:

ls -l <OUTPUT_DIR>/UDP/ | wc -l

which returns the total number of CoNLL-U files created (each file corresponds to a document).

Example output directory UDP 📁 contains per-document CoNLL-U files.

[!NOTE] Chunking and page boundaries. chunk.py📎 splits text on OCR line boundaries (not raw whitespace), preserving the newline-separated structure of the source CSV so that UDPipe receives proper sentence-boundary hints between lines. When a document spans multiple chunks, call_udpipe.py📎 merges them into a single CoNLL-U and injects a # page_break = true comment immediately before every sentence that began a new page in its source chunk. All downstream scripts (call_nametag.py📎, summarize_nt_udp.py📎, teitok_alto.py📎) recognise this marker alongside the legacy # sent_id = 1 page-reset convention, so both single-chunk and multi-chunk files are handled transparently.

[!TIP] You can launch the next step when a portion of CoNLL-U files are ready, without waiting for the entire input collection to finish. You will have to relaunch the next step after all CoNLL-U files are ready to process the files created after the previous run began.

3. NameTag Processing (NER tags)

Takes the valid CoNLL-U files and passes them through the NameTag API ³ to annotate Named Entities (NE) directly into the syntax trees.

./api_3_nt.sh

Input: OUTPUT_DIR/UDP/*.conllu (Intermediate per-document CoNLL-U files).
Output: OUTPUT_DIR/NE/*/*.tsv (NE annotated per-page files)

Run the following command to see how many documents have been processed into TSV files:

ls -l OUTPUT_DIR/NE | wc -l

which returns the total number of directories created (each subfolder corresponds to a document).

Example output directory NE 📁 contains per-page TSV files with NE annotations, where the NE tags follow the CNEC 2.0 standard ⁴ which is used in the Czech Nametag model.

4. Generate Statistics

./api_4_stats.sh

Inputs and Outputs

Input 1: OUTPUT_DIR/UDP/*.conllu — Per-document CoNLL-U files containing morphology and syntax.
Input 2: OUTPUT_DIR/NE/*/*.tsv — Per-page TSV files containing Named Entity annotations.
Input 3 (Optional): INPUT_ALTO_DIR/*.alto.xml — Source ALTO XML files used during TEITOK conversion to provide spatial bounding box coordinates for each token.
Input 4 (Optional): INPUT_PAGES_DIR/<doc_id>-N.png — Per-page facsimile images. When specified, the pipeline dynamically extracts pixel boundaries from the headers to compute scaling transformations (sx, sy). If omitted, coordinates are safely translated and aligned at a native 1.0 scale factor.
Output 1: OUTPUT_DIR/summary_ne_counts.csv — Global table of aggregated Named Entity statistics across all documents.
Output 2: OUTPUT_DIR/UDP_NE/<doc_id>/<doc_id>.csv — Per-document CSV tables with tokens, lemmas, and human-readable NE explanations.
Output 3 (Optional): OUTPUT_DIR/UDP_NE/<doc_id>/<doc_id>.conllu — Final CoNLL-U files with NER tags enriched in the MISC column.
Output 4 (Optional): OUTPUT_DIR/TEITOK/<doc_id>.teitok.xml — TEITOK-style TEI XML files ready for the flexiconv converter and facsimile viewing (see below).

The behavior of this step is controlled by boolean flags in your config_api.txt:

Variable	Description	Default
`SAVE_CONLLU_NE`	Keep the enriched CoNLL-U with NER in the `MISC` field.	`true`
`SAVE_CSV`	Write the token-level summary CSV per document.	`true`
`SAVE_TEITOK`	Write TEITOK-style TEI XML with bounding boxes and NER spans. When `INPUT_ALTO_DIR` is not set a warning is emitted and TEITOK XML is still produced without bboxes. If `INPUT_ALTO_DIR` is set but the path does not exist, the step exits with an error.	`true`
`INPUT_PAGES_DIR`	Directory of per-page images (`<doc_id>-N.png`). When set, bbox coordinates are scaled to match the actual PNG resolution. Leave empty to write raw ALTO pixel values.	(empty)

ALTO-to-TEITOK XML Generation and Coordinate Alignment

When SAVE_TEITOK=true, teitok_alto.py 📎 reads and processes the internal spatial hierarchy of your ALTO source specifications.

Offset Alignment (Resolving Layout Shifting):

To neutralize this error without modifying binary assets or re-cropping, the script automatically parses page-level <PrintSpace> properties from the ALTO structure:

<PrintSpace HEIGHT="3263" WIDTH="2027" VPOS="80" HPOS="297">

$$\text{Scaled Coordinate} = \text{round}((\text{Absolute Coordinate} - \text{Offset}) \times \text{Scale Factor})$$

Dynamic Scale Calculations:

Companion Image Present (Tier 1): If INPUT_PAGES_DIR is set and matching images exist, the tool safely reads binary file headers without invoking bloated third-party imaging dependencies. Ratios are resolved by evaluating layout sizes against image shapes (sx = img_width / alto_width).
User-set DPI (Tier 2): If no image is available, scale is derived directly from the ALTO <MeasurementUnit> (inch1200, mm10, or pixel) mapped against the environment variables IMAGE_DPI and ALTO_DPI.
Native Processing (Fallback): If no image and no DPI is provided, the tool calibrates positions through native PrintSpace logic but maintains a standard 1.0 scale factor.

Future direction: Relative / resolution-independent coordinates are the preferred long-term direction (pending TEITOK-team confirmation).

Fixing Bounding Box Alignments (Practical Guide)

Here is exactly how you would use the integrated tools to solve this problem.

Method 1: The Quick API Fix (Best for single files or web integrations)

Since the pipeline now includes a dedicated FastAPI service, you don't even need to write a script. You can just send your misaligned XML to the /rescale endpoint.

Open your terminal and run a simple curl command, explicitly telling the API the exact dimensions of your target image and requesting the output as an XML file instead of the default JSON metadata:

curl -X POST "http://localhost:8000/rescale" \
     -F "file=@CTX000000001.teitok.xml" \
     -F "width=1200" \
     -F "height=1800" \
     -F "format=xml" \
     -o CTX000000001.rescaled.teitok.xml

Method 2: The Command-Line Batch Process (Best for whole directories)

If you know your web images are exactly 50% the size of your original scans (a scale factor of 0.5), you can process the entire directory at once:

python3 fix_teitok_bboxes.py -i /path/to/my/teitok_folder/ --sx 0.5 --sy 0.5

Alternatively, if your original ALTO OCR data was in millimeters (mm10) and you need to target a standard 72 DPI screen resolution, the script can handle that math directly:

python3 fix_teitok_bboxes.py -i my_document.teitok.xml --unit mm10 --dpi 72

python3 fix_teitok_bboxes.py -i my_document.teitok.xml --dx -50 --dy -20

[!NOTE] When a token's matched ALTO strings span more than one page (a rare OCR edge case near page boundaries), a warning is printed to stderr identifying the token and the conflicting page indices. The first matched page is used for the bbox assignment in that case.

The structural and spatial hierarchy from the ALTO file is strictly preserved in the generated TEITOK XML:

Tokens: Matched coordinates are written to each <tok> element as @bbox="x1 y1 x2 y2" (absolute pixel coordinates in TEITOK's hOCR-derived format). Each token also carries @type="w" (word) or @type="pc" (punctuation character) derived from UDPipe's UPOS tag.
Lines: ALTO <TextLine> elements are preserved via <lb> (line break) tags, which also include their own @bbox spatial coordinates.
Blocks: Text blocks are encapsulated within <div type="MarginTextZone-P"> containers, satisfying the core ATRIUM guidelines for classified text zones.
Graphics: Non-text elements like Illustration and GraphicalElement blocks are parsed and appended to their respective pages as <figure> tags with strict bounding boxes.
Pages: Page boundaries are marked with <pb n="N" id="..." facs="..."/> elements pointing to the specific document surface.

[!NOTE] Thanks to the sequence matching approach, the script achieves near-perfect spatial alignment between NLP tokens and OCR coordinates, drastically improving upon older greedy matching methods that would break on minor character variations. Alignment statistics (matched vs. total tokens) are printed to the console per document.

ls OUTPUT_DIR/UDP_NE | wc -l

which returns the total number of created files, both .csv and .conllu corresponding to specific documents.

ls OUTPUT_DIR/UDP_NE/*/*.csv | wc -l

returns number of documents processed into tables

ls OUTPUT_DIR/TEITOK/*.xml | wc -l

returns number of recorded .teitok.xml documents.

Example summary table: summary_ne_counts.csv (produced by a real run; not committed — see the note below).

Example output directory UDP_NE 📁 contains per-document CSV tables with NE tags and UDPipe feature columns, plus CoNLL-U files with NE annotations in per-document manner.

Example output directory TEITOK 📁 contains per-document TEITOK XML files combining UD linguistic annotations and NER spans with bounding boxes aligned from the source ALTO XML.

Output Structure

After completing the pipeline, your working and output directories will be organized as follows:

TEMP/
├── CHUNKS/
│   └── ...
├── nametag_response_docname1.conllu.json
└── ...

AND

<OUTPUT_DIR>
├── UDP_NE/
│   ├── <doc_id>
│   │   ├── <doc_id>.csv
│   │   └── <doc_id>.conllu
│   ├── <doc_id>
│   │   ├── <doc_id>.csv
│   │   └── <doc_id>.conllu
│   └── ...
├── UDP/
│   ├── <doc_id>.conllu
│   ├── <doc_id>.conllu
│   └── ...
├── TEITOK/
│   ├── <doc_id>.teitok.xml
│   ├── <doc_id>.teitok.xml
│   └── ...
├── NE/
│   ├── <doc_id>
│   │   ├── <doc_id>-<page_num>.tsv
│   │   └── ...
│   ├── <doc_id>
│   │   ├── <doc_id>-<page_num>.tsv
│   │   └── ...
│   └── ...
├── altos/
│   ├── <doc_id>.alto.xml
│   └── ...
├── pages/
│   ├── <doc_id>-1.png
│   ├── <doc_id>-2.png
│   └── ...
├── processing.log
├── summary_ne_counts.csv
└── manifest.tsv

[!NOTE] Now you can delete UDP/ from <OUTPUT_DIR>/ if you no longer need the raw CoNLL-U files. The final CoNLL-U files with NER features are in <OUTPUT_DIR>/UDP_NE/.

If you do not plan to rerun any part of the pipeline, you can also delete the entire TEMP/ directory including manifest.tsv 📎.

EXTRA: Extract Keywords (KER / YAKE / KeyBERT)

[!NOTE] This is an optional step in NLP enrichment of your data. It can give a fast thematic overview of the whole collection and works best when UDPipe lemmas (output of Step 2) are available. Three extraction backends are provided; choose the one that best fits your environment and quality requirements.

Extract keywords 🔎 from your documents by running keywords.py on a directory of CoNLL-U files produced by Step 2.

Configuration Priority

The keyword extraction script uses a three-tier configuration hierarchy (from highest to lowest priority):

Command-line flags (e.g., -m yake, -w 3) always override everything else.
kw_config.txt (the [DEFAULTS] section) is read automatically if placed next to the script.
Hardcoded fallbacks are used if no config file or flags are provided.

This means if you configure your settings in kw_config.txt, you can simply run:

python3 keywords.py

Backends

Flag value	Method	Dependencies	Score semantics	Best for
`legacy`	Original KER — NOUN/PROPN/ADJ lemma frequency	none (stdlib only)	raw occurrence count	reproducing original ATRIUM results
`yake` (default)	YAKE — unsupervised statistical, CPU-only	`pip install yake`	normalised inverse YAKE score, [0, 1]	fast CPU runs, no model download
`keybert`	KeyBERT — embedding-based, GPU-accelerated	`pip install keybert sentence-transformers`	cosine similarity, [0, 1]	highest semantic quality, GPU recommended

You can override any kw_config.txt setting via the command line:

python3 keywords.py -i <input_dir> -m <method> -l <lang> -w <integer> \
                    -n <integer> -d <output_dir> -o <output_file>.csv

All available flags:

Flag	Long form	Default in `kw_config.txt`	Description
`-i`	`--input_dir`	`data_samples/UDP`	CoNLL-U directory to process
`-m`	`--method`	`yake`	Backend: `legacy`, `yake`, or `keybert`
`-l`	`--lang`	`cs`	Language code for YAKE stopwords (`cs`, `en`, `de`, …). Ignored by `legacy` and `keybert`
`-w`	`--max_words`	`3`	Maximum words per keyword phrase (n-gram upper bound)
`-n`	`--num_keywords`	`20`	Number of keywords to extract per document
`-d`	`--per_doc_out_dir`	`data_samples/KW_PER_DOC`	Output directory for per-document CSV files
`-o`	`--output_file`	`keywords_summary.csv`	Master keywords CSV
	`--keybert-model`	`paraphrase-multilingual-MiniLM-L12-v2`	Sentence-Transformer model name (KeyBERT only)
	`--no-mmr`	(False)	Disable Maximal Marginal Relevance diversification (KeyBERT only)
	`--diversity`	`0.5`	MMR diversity parameter, 0 = max relevance → 1 = max diversity (KeyBERT only)
	`--workers`	`0` (Auto / CPU count)	Parallel worker processes. Auto-forced to 1 for KeyBERT + GPU

Examples:

YAKE — Czech, up to 3-word phrases, 20 keywords per document (default)

python3 keywords.py -i OUTPUT_DIR/UDP -m yake -l cs -w 3 -n 20 \
        -o keywords_summary.csv -d KW_PER_DOC

KeyBERT — multilingual model, GPU-accelerated

python3 keywords.py -i OUTPUT_DIR/UDP -m keybert -w 3 -n 20 \
        --keybert-model paraphrase-multilingual-MiniLM-L12-v2 \
        -o keywords_summary.csv -d KW_PER_DOC

Legacy KER — (English/Czech) original ATRIUM lemma-frequency approach, no extra dependencies

python3 keywords.py -i OUTPUT_DIR/UDP -m legacy -n 20 \
        -o keywords_summary.csv -d KW_PER_DOC

[!WARNING] For KeyBERT with a GPU, the script automatically forces --workers 1 to prevent competing CUDA context initialisation across subprocesses. On CPU, any worker count is safe.

Inputs and outputs

Input: Directory of per-document CoNLL-U files from Step 2.
Output 1: Master table with keywords per document (e.g., keywords_summary.csv).
Output 2: Per-document CSV files (e.g., KW_PER_DOC/).

KW_PER_DOC/
├── <docname1>_keywords.csv
├── <docname2>_keywords.csv
└── ...

Score interpretation by backend

legacy — raw lemma count; higher = more frequent in the document. Examples in directory: KW_PER_DOC_L 📂 and summary file kw_summary_l.csv 📎.

Score range	Interpretation
1–5	Common functional nouns, low informativeness
5–20	Topic-representative vocabulary
> 20	Dominant terms, likely named entities or domain headings

yake — normalised inverse YAKE score, [0, 1] per document. Examples in directory: KW_PER_DOC_Y 📂 and summary file kw_summary_y.csv 📎.

Score range	Semantic category	Interpretation
0.0–0.2	Noise floor	Common words, low local relevance
0.2–0.6	Context layer	General vocabulary defining the broad topic
0.6–0.9	Topic layer	Specific nouns and verbs central to the text
0.9–1.0	Entity layer	Rare terms, neologisms, named entities

keybert — cosine similarity to document centroid, [0, 1]. Examples in directory: KW_PER_DOC_KB 📂 and summary file kw_summary_kb.csv 📎.

Score range	Interpretation
< 0.3	Weakly related phrases
0.3–0.6	Contextually relevant terms
> 0.6	Highly representative keyphrases

EXTRA: Converting Other Input Formats with flexiconv

[!NOTE] This section is relevant when your documents originate from an OCR or digitisation pipeline that does not produce ALTO XML — for example, PAGE XML, hOCR, plain-text exports, or proprietary formats. If you already have ALTO XML, the pipeline generates TEITOK XML natively via api_4_stats.sh (see above).

What is flexiconv?

  Your input format           flexiconv             Unified output
  ─────────────────    ─────────────────────────   ─────────────────
  PAGE XML          ─┐
  hOCR              ─┤──► flexiconv ──────────────► .teitok.xml ──► TEITOK platform
  plain text + CSV  ─┤                                            ──► this pipeline
  other OCR output  ─┘                                               (NER, KWs, ...)

When to use flexiconv

Use flexiconv before running this pipeline when:

Your collection was OCR-processed with a tool that outputs PAGE XML (e.g. Transkribus, OCRopus, kraken).
Your layout data is in hOCR format (used by Tesseract and some ABBYY exports).
You have structured text with positional metadata but no standard bounding-box format.
You received digitised material from a partner institution using a format not natively supported by teitok_alto.py.

How to use flexiconv

Clone and install the tool:

git clone [https://github.com/ufal/flexiconv.git](https://github.com/ufal/flexiconv.git)
cd flexiconv
pip install -r requirements.txt

Run the conversion on your input files:

python flexiconv.py \
    --input-dir  /path/to/your/source/documents \
    --input-fmt  page-xml \          # or: hocr, plain, ...
    --output-dir /path/to/teitok_out \
    --output-fmt teitok

Refer to the flexiconv documentation for the full list of supported --input-fmt values and format-specific options.

Continue with this pipeline using the converted TEITOK XML files. At this point your documents already have layout structure and bounding boxes embedded — the NLP enrichment steps (UDPipe morphology, NameTag NER, keyword extraction) can be applied on top via the scripts in this repository.

[!TIP] If your format is not yet supported by flexiconv, please open an issue on the flexiconv GitHub repository. The tool is actively developed within the ATRIUM project and new format adapters are added regularly.

EXTRA: LLM Semantic Enrichment (Vocabulary Mapping)

[!NOTE] This is an advanced, optional step. It runs a local Large Language Model to semantically analyse each text line and map it to the controlled TEATER/AMCR archaeological vocabulary. Two inference backends are supported: transformers (HuggingFace + BnB 4-bit, single GPU, models ≤ 31 B) and vllm (multi-GPU, Automatic Prefix Caching, native guided JSON decoding, models ≥ 70 B or any multi-GPU node).

⚙️ Configuration (llm_config.txt 📎)

The pipeline reads all runtime parameters from llm_config.txt in the repository root. The minimum required change is MODEL_KEY; every other key has a sensible default.

# Single-GPU (BACKEND=transformers): qwen-3.6-27b-it | gemma-4-31b-it | qwen3-14b |
#                                    qwen-3.5-9b-it | qwen3-8b | qwen2.5-14b-awq |
#                                    qwen2.5-7b | gemma-3-12b-it
# MoE / GGUF (single GPU):           gemma-4-26b-moe-gguf | qwen-3.6-35b-moe
# Multi-GPU (BACKEND=vllm):          qwen3-235b-a22b-fp8 | deepseek-v3 | llama4-maverick | llama3.1-70b
MODEL_KEY=qwen-3.6-27b-it

# Only needed for gated models: gemma-4-*, llama4-maverick, llama3.1-70b
# HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxx

INPUT_DIR=data_samples/DOC_LINE_CATEG
OUTPUT_DIR=data_samples/KW_PER_DOC_LLM
VOCAB_PATH=data_samples/vocab/union_nested.json
PARADATA_DIR=paradata

# Attach the surviving vocabulary term's source record id(s) to each enrichment as
# teater_category_ids (issue #6, M7). Kept behind a switch since it was agreed to be
# reversible: "list them now and drop it if it will create some issues."
EMIT_CATEGORY_IDS=true

INCLUDE_NON_TEXT=true
MIN_CHAR_COUNT=3
MIN_CHAR_NON_TEXT=8
MIN_ALPHA_RATIO_NON_TEXT=0.4

# ── System prompt ─────────────────────────────────────────────────────────────
# The instruction text lives in prompts/system_prompt.txt as [[named blocks]]; these
# flags choose which of them render. The run banner prints the resulting on/off list,
# so a log always says what the model was told. See prompts/RUNBOOK.md.
PROMPT_TEMPLATE=prompts/system_prompt.txt
PROMPT_TASK_EXTRACT=true
PROMPT_TASK_SELECT=true
PROMPT_METATEXT_RULE=true
PROMPT_OCR_NORMALISATION=true
PROMPT_EXACT_TERM=true
PROMPT_EXAMPLES=true

# The one three-way switch: strict | preference | off. Paired with
# taxonomy_config.json's geo_guardrail.active — vocab_build.py refuses a build where
# the two disagree.
PROMPT_GEO_GUARDRAIL=preference

# Term-list layout: facet_sub | facet | flat. Same terms, same truncation; only the
# headers move.
PROMPT_VOCAB_GROUPING=facet_sub

# ── Inference parameter overrides ─────────────────────────────────────────────
# ALL of these are COMMENTED OUT in the shipped file. Backend, GPU count, memory
# utilisation, batch size and context cap are resolved automatically from the model
# registry in llm_utils.py; the startup log prints every effective value next to
# where it came from (← llm_config.txt / model default / global default). Uncomment
# a line only to deviate from the model's recommended configuration.
# BACKEND=vllm                 # transformers (single GPU, ≤ 31 B) | vllm (multi-GPU)
# TENSOR_PARALLEL_SIZE=8       # GPUs to shard across (vLLM only)
# GPU_MEMORY_UTILIZATION=0.88  # Fraction of each GPU's VRAM for the KV cache
# VLLM_BATCH_SIZE=8            # Lines per generate() call
# MAX_MODEL_LEN=16384          # Cap the context window to reduce KV-cache pressure
# CPU_OFFLOAD_GB=0             # Weights to keep in CPU RAM when VRAM is short
# GUIDED_DECODING_BACKEND=xgrammar
# ENABLE_PREFIX_CACHING=false  # Not recommended; reduces throughput

The nine PROMPT_* keys are the prompt's whole configuration surface; prompts/RUNBOOK.md 📎 documents each block, what it costs, and which pairings are unsafe to change alone.

🗂 Workflow

1. Vocabulary Harvesting (vocab_build.py 📎)

The vocabulary is built in two stages, and only the first needs the internet:

harvest (network)   →   FLAT artifacts    →   nest (pure)    →   NESTED artifacts
vocab_sources.py        *_flat.{json,csv}     vocab_manager      *_nested.json

vocab_sources.py 📎 harvests two controlled vocabularies:

Source	How	What comes back
AMCR heslář	OAI-PMH, `api.aiscr.cz/2.2/oai?set=heslo`	Czech–English pairs plus `ident_cely`, `nazev_heslare` (which of the ~50 controlled lists the term belongs to), `popis`, `zkratka`, `razeni`, broader terms and SKOS mappings
TEATER thesaurus	the 12 pinned `import_*.json` files in `ARUP-CAS/aiscr-teater`, or live `teater.aiscr.cz/api/export`	4 134 concepts in 12 branches, trilingual labels, scope notes, and the real broader/narrower hierarchy

Every vocabulary decision is a config edit, not a code change. The two JSON files are the whole surface a domain reviewer needs; nothing below requires touching Python:

In `taxonomy_config.json` → `_settings`	Decides
`heslar_map`, `teater_branch_map`	which facet a whole AMCR list or TEATER branch lands in, or `__exclude__`
`_exclusions`	why each exclusion stands, and whether it is `settled` or still open (`open_geo_ethnic` / `open_other`)
`geo_guardrail`	whether the prompt's "never select a country/language/region name" clause is in force, and which rules it reaches
`nested_keep`	which harvested keys reach the prompt payload
`admin_stop_words`	what sorts to the back of a facet, and so what survives prompt truncation
`composite_separators`	what splits a composite `X/Y` label
`tie_break`, per-facet `priority`	facet order — load-bearing, since the prompt truncates a prefix

In `taxonomy_overrides.json`, per `(source, id)`	Decides
`facet`	one term's facet, including `"__exclude__"` to drop a single term from a list worth keeping
`sub`	one term's sub-header — otherwise a moved term keeps the header of the list it left
`qualifier_cs`	pull a confirmed homonym out of its dedup group as `"<cs> (<qualifier>)"`
`same_as` / `same_as_suppress`	add or drop a composite/component equivalence link; neither changes what the prompt offers

Reading the prompt needs neither a GPU nor the model stack — prompt_template.py imports nothing outside the standard library, so these run in a bare checkout:

python3 prompt_template.py --blocks                  # which rules are on, and what each costs
python3 prompt_template.py --preview                 # the instruction text, term list elided
python3 prompt_template.py --full  > prompt.txt      # the whole prompt, all 4 718 terms
python3 prompt_template.py --diff PROMPT_GEO_GUARDRAIL=strict \
                                 PROMPT_GEO_GUARDRAIL=preference
python3 prompt_template.py --write                   # regenerate the committed sheets
python3 prompt_template.py --check                   # exit 1 if a sheet is out of date

# stage 1 + 2, needs network access to aiscr.cz
python3 vocab_build.py --source both --stats

# stage 2 only: re-nest from the committed flat files after editing the taxonomy.
# Pure, offline, sub-second — this is the loop for tuning the taxonomy.
python3 vocab_build.py --from-flat --stats
python3 vocab_build.py --from-flat --check     # exit 1 if the artifacts would change

If this machine cannot reach aiscr.cz, run the Vocabulary Refresh workflow (.github/workflows/vocab-refresh.yml) — a hosted runner harvests and uploads the artifacts.

[!NOTE] The nested files are deliberately not written with sort_keys=True. Theme order is priority-descending and load-bearing: build_system_prompt() iterates the file in insertion order and truncates a prefix of the resulting term list, so alphabetising the keys would silently change which themes survive a tight context budget. Determinism comes from the explicit priority ordering plus a (boilerplate, razeni, label) sort within each theme. Provenance lives in a sidecar *.meta.json, not inline — every consumer reads the nested file as {theme: terms}, so an inline _meta key would be rendered to the model as a phantom theme.

python3 vocab_manager.py still works and still performs the legacy AMCR-only sync.

Reviewing the vocabulary

vocab_review.py 📎 — eight sheets, offline and pure, built from the committed *_flat.json plus the taxonomy config:

python3 vocab_review.py --all            # all eight, into data_samples/vocab/
python3 vocab_review.py --collisions     # collision_review.csv        same-label groups (M8/M13)
python3 vocab_review.py --composites     # composite_pairs.csv         "X/Y" vs standalone X, Y
python3 vocab_review.py --exclusions     # exclusion_impact.csv        what each exclusion costs
python3 vocab_review.py --subbranches    # teater_subbranch_impact.csv the same, one level finer
python3 vocab_review.py --reinstate      # reinstatement_preview.csv   usable count + token delta
python3 vocab_review.py --specificity    # specificity_pairs.csv       term offered under its own parent
python3 vocab_review.py --budget         # context_budget.csv          what survives each window
python3 vocab_review.py --census         # facet_census.csv            what is in each facet, and its cost

corpus_review.py 📎 — three evidence sheets matching the vocabulary against real report text by UDPipe lemma, which is the evidence standard for keeping or dropping a branch:

python3 corpus_review.py --all           # corpus_term_evidence.csv, corpus_branch_evidence.csv,
                                         # gold_workbook.csv (+ corpus_review.meta.json)

2. LLM Inference Pipeline (llm_run.py 📎)

[!TIP] All model-loading logic, constrained-decoding helpers, and prompt templates live in llm_utils.py 📎 and are shared between both backends.

# Transformers backend (default)
python3 llm_run.py

# Custom config file
python3 llm_run.py my_config.txt

For multi-GPU runs (vLLM backend):

# 1. Edit llm_config.txt:
#    BACKEND=vllm
#    MODEL_KEY=qwen3-235b-a22b-fp8
#    TENSOR_PARALLEL_SIZE=2
#    ENABLE_PREFIX_CACHING=true
python3 llm_run.py

🖥 Model Registry

The built-in registry in llm_utils.py covers the full range of supported models. All VRAM figures assume BnB 4-bit for the transformers backend and FP8/BF16 for vLLM.

Single-GPU — `BACKEND=transformers` (or `BACKEND=vllm`)

Registry key	Model	Size	Context	Est. VRAM	Notes
`qwen-3.6-27b-it`	Qwen/Qwen3.6-27B ⁶	27 B dense	262 k	~18 GB	Default. Best accuracy/VRAM ratio on a single GPU.
`gemma-4-31b-it`	google/gemma-4-31B-it ⁷	31 B dense	256 k	~21 GB	Highest single-GPU accuracy. Gated — `HF_TOKEN` required.
`qwen3-14b`	OpenPipe/Qwen3-14B-Instruct ⁸	14 B dense	128 k	~9 GB	Good baseline; thinking mode suppressed automatically.
`qwen-3.5-9b-it`	Qwen/Qwen3.5-9B ⁹	9 B dense	262 k	~6 GB	Entry-level (8 GB VRAM).
`qwen3-8b`	Qwen/Qwen3-8B ¹⁰	8 B dense	128 k	~16 GB	BF16 (no 4-bit); straightforward baseline.
`qwen2.5-14b-awq`	Qwen/Qwen2.5-14B-Instruct-AWQ ¹¹	14 B AWQ	128 k	~9 GB	Pre-quantized; fast on NVIDIA GPUs.
`qwen2.5-7b`	Qwen/Qwen2.5-7B-Instruct ¹²	7 B dense	32 k	~14 GB	BF16; short context window.
`gemma-3-12b-it`	google/gemma-3-12b-it ¹³	12 B dense	128 k	~8 GB	Good bilingual extraction. Gated.

MoE models — GGUF / llama.cpp fallback (any GPU, any VRAM)

Registry key	Model	Active params	Context	Notes
`gemma-4-26b-moe-gguf`	bartowski/google_gemma-4-26B-A4B-it-GGUF	4 B	8 k	BnB 4-bit unsupported (fused experts). Q4_K_M quantization via llama.cpp.

MoE models — `BACKEND=vllm` (single GPU or multi-GPU)

Registry key	Model	Active params	Context	Notes
`qwen-3.6-35b-moe`	Qwen/Qwen3.6-35B-A3B ¹⁴	3 B	262 k	35 B total / 3 B active. Single GPU usually fits.
`gemma-4-26b-moe`	google/gemma-4-26B-A4B-it ¹⁵	4 B	256 k	26 B total / 4 B active. Gated.
`gemma-4-26b-moe-awq`	google/gemma-4-26B-A4B-it ¹⁵	4 B	256 k	AWQ-quantised variant of `gemma-4-26b-moe`. Gated.

Large models — `BACKEND=vllm`, `TENSOR_PARALLEL_SIZE ≥ 2`

Registry key	Model	Total / Active	Context	Rec. TP	Notes
`qwen3-235b-a22b-fp8`	Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 ¹⁶	235 B / 22 B	128 k	2	Recommended for 144 GB / 200 GB nodes. Native FP8 (~117 GB loaded).
`qwen3-235b-a22b`	Qwen/Qwen3-235B-A22B-Instruct-2507 ¹⁶	235 B / 22 B	128 k	2	BF16 variant — heavier than FP8.
`deepseek-v3`	deepseek-ai/DeepSeek-V3 ¹⁷	671 B MoE / —	128 k	4	FP8 official checkpoint available. 4×80 GB minimum.
`llama4-maverick`	meta-llama/Llama-4-Maverick-17B-128E-Instruct ¹⁸	128 experts / 17 B active	1 M	2	Multimodal. 1 M token context. Gated — `HF_TOKEN` required.
`llama3.1-70b`	meta-llama/Meta-Llama-3.1-70B-Instruct ¹⁹	70 B dense / —	128 k	2	Also works with `transformers` + 4-bit on 2×40 GB. Gated.

[!TIP] Automatic Prefix Caching (APC) — enabled by default for the vLLM backend (ENABLE_PREFIX_CACHING=true). The system prompt (which embeds the full TEATER vocabulary) is computed once per run; its KV-cache is reused across every line in every document. This is the primary throughput multiplier: on a 500-line document the vocabulary forward pass happens once instead of 500 times. APC also removes the need to truncate the vocabulary to fit the token budget — the full thematic dictionary is injected when APC is active.

📁 Inputs and Outputs

Input: DOC_LINE_CATEG/*.csv (contains file_id, page_num, line_num, categ, quality_score, and raw text).
Output: KW_PER_DOC_LLM_<model_suffix>/*_enriched.json — one file per document, containing an array of JSON objects that merge CSV metadata with the LLM's semantic extraction.
Abort sidecar: KW_PER_DOC_LLM_<model_suffix>/*_enriched.abort.json — written alongside the main output only when a document is abandoned after 10 consecutive inference errors. Its presence is the canonical signal that the corresponding JSON file contains partial results.

Example output record:

{
  "file_id": "CTX195603828",
  "page": 1,
  "line": 14,
  "categ": "Text",
  "quality_score": 0.98,
  "original_text": "Výzkum odhalil základy gotického kostela ze 14. století.",
  "enrichment": {
    "extracted_keywords_cs": ["základy", "gotický kostel"],
    "extracted_keywords_en": ["foundations", "gothic church"],
    "teater_category": "kostel",
    "teater_category_ids": [
      { "source": "amcr", "id": "HES-000021" },
      { "source": "amcr", "id": "HES-000465" },
      { "source": "teater", "id": "1333" }
    ],
    "confidence_score": 0.95
  }
}

Abort sidecar format (*_enriched.abort.json):

{
  "aborted": true,
  "abort_reason": "10 consecutive inference errors",
  "processed_before_abort": 42,
  "errors_before_abort": 10,
  "timestamp_utc": "2026-05-20T09:14:33"
}

[!NOTE] None of the per-model output sets below is committed to this repository — each is the real output of a real run against the full report corpus, not a sample bundled with the code (same reason data_samples/DOC_LINE_CATEG/ itself holds only three synthetic demo documents). The directory names are the OUTPUT_DIR a local run with that MODEL_KEY produces; the footnote on each is the model card.

Output examples per model (directory names, not links — see the note above):

KW_PER_DOC_LLM_qwen3_14b by Qwen 3-14B ⁸
KW_PER_DOC_LLM_qwen25_14b_awq by Qwen 2.5-14B AWQ ¹¹
KW_PER_DOC_LLM_gemma_3_12b_it by Gemma 3-12B-IT ¹³
KW_PER_DOC_LLM_qwen_36_27b_it by Qwen 3.6-27B-IT ⁶
KW_PER_DOC_LLM_gemma_4_31b_it by Gemma 4-31B-IT ⁷
KW_PER_DOC_LLM_qwen_35_9b_it by Qwen 3.5-9B-IT ⁹
KW_PER_DOC_LLM_llama31_70b by LLaMA 3.1-70B ¹⁹
KW_PER_DOC_LLM_qwen3_8b by Qwen 3-8B ¹⁰

Pending (sample runs in progress):

KW_PER_DOC_LLM_qwen_36_35b_moe by Qwen 3.6-35B-MoE ¹⁴
KW_PER_DOC_LLM_gemma_4_26b_a4b_it by Gemma 4-26B-A4B-IT ¹⁵

Archived (unsuccessful — evaluation notes in issue #6; would have been under archived_KW_PER_DOC_LLM/):

KW_PER_DOC_LLM_mistral_nemo_12b by Mistral Nemo 12B ²⁰
KW_PER_DOC_LLM_aya_expanse_8b by Aya Expanse 8B ²¹
KW_PER_DOC_LLM_bielik_11b_v30 by Bielik 11B v3.0 ²²
KW_PER_DOC_LLM_llama31_8b by LLaMA 3.1-8B ²³
KW_PER_DOC_LLM_ministral_3_14b by Ministral 3-14B ²⁴
KW_PER_DOC_LLM_qwen3_8b (early run) by Qwen 3-8B ¹⁰
KW_PER_DOC_LLM_qwen25_7b by Qwen 2.5-7B ¹²

📊 Paradata Integration

Just like the main shell-script pipelines, LLM enrichment natively hooks into atrium_paradata.py and automatically logs:

Full snapshot of llm_config.txt 📎 and quality-filter settings.
Which vocabulary build was used — read from the *.meta.json beside the artifact: tool version, term count, the sha256 of both taxonomy files, and each source's record count and pinned ref (TEATER's harvest commit). A run is reproducible only if the vocabulary it saw is identifiable, and every placement decision is a function of those two sha256s.
Both vocabulary sources as licence components. The AMCR heslář and the TEATER thesaurus are CC BY-NC 4.0 and declared conditional in para_config.txt 📎, so they constrain a run's effective licence only when log_component() names them. Logged per source actually present in the build — an AMCR-only artifact does not claim it used TEATER data.
Total processed lines (json success events).
Per-line tracking of filter skips (skipped_filter), inference faults (skipped_error), and already-completed files (already_exists).
Abort events — when a document is abandoned after 10 consecutive inference errors, the paradata entry records the abort reason alongside the count of lines processed before the failure. A sidecar *.abort.json file is also written next to the (partial) output JSON for easy programmatic detection. The resulting logs are dropped into the specified PARADATA_DIR alongside the other pipeline execution records.

EXTRA: REST API Service

The pipeline now includes a fully-featured FastAPI REST service that exposes the core NLP enrichment and rescaling functionalities over HTTP.

Single-file enrichment: Upload CSV, XLSX, or plain text to the /enrich endpoint and receive a combined JSON envelope (or ZIP workspace) with TEITOK XML, keywords, paradata, and NER summaries.
Coordinate Rescaling: Use the /rescale endpoint to align XML spatial coordinates to specific target image resolutions directly over the network.
Job Management: Background processing for larger documents with a asynchronous /jobs queue.

For complete setup instructions, payload examples, and endpoint documentation, refer to the Service README.

Paradata Logs

Every pipeline script records structured provenance metadata through atrium_paradata.py 📎. Two complementary log surfaces are produced after a run:

`<OUTPUT_DIR>/paradata/` — structured run logs 📂

Each of the four pipeline scripts produces one JSON file here, named with the pattern:

YYMMDD-HHmmss_nlp-enrich.json

The declared output types per stage are:

Script	Types recorded
`api_1_manifest.sh`	`tsv` (one entry per input CSV/XLSX processed into the manifest)
`api_2_udp.sh`	`conllu` (one per document)
`api_3_nt.sh`	`tsv` (one per page — count reflects individual page TSV files)
`api_4_stats.sh`	`csv` always; `conllu` when `SAVE_CONLLU_NE=true`; `xml` when `SAVE_TEITOK=true`
`keywords.py`	`csv_per_doc` (one per document keyword CSV) and `csv_summary_row` (one summary row per document)

[!NOTE] When resuming an interrupted run (steps 2–4 skip already-finished documents via [ -f "$out" ] && continue), the resumed documents are not re-counted in the paradata JSON. The input_files_total field still reflects the full manifest, so skipped_files + successfully_processed will be less than input_files_total for partial runs. This is expected behaviour; the difference represents the documents carried over from a previous invocation.

[!NOTE] Paradata state files. While a pipeline script is running, atrium_paradata.py stores intermediate state in a plain-text JSON file inside <OUTPUT_DIR>/paradata/ (named .state_<runid>_<program>.json). This file is automatically removed when the script completes. Because it is plain JSON it can be inspected with any text editor if a run is interrupted unexpectedly.

`<OUTPUT_DIR>/processing.log` — human-readable runtime log 📄

[2026-01-15 09:42:11] [WARN] UDPipe failed (HTTP 503). Retrying in 2s…
[2026-01-15 09:42:14] [ERR]  UDPipe failed permanently after 5 attempts.

This file accumulates across reruns; it is the first place to check when a document appears in skipped_files_detail but the reason is terse.

`TEMP/` — intermediate working files 📂

TEMP/ (set by WORK_DIR in config_api.txt 📎) holds transient artefacts that are only needed during processing and can be deleted once the full pipeline has completed successfully:

TEMP/
├── CHUNKS/
│   ├── <doc_id>/
│   │   ├── chunk_0.txt      # OCR-line-preserving text fragment sent to UDPipe
│   │   ├── chunk_1.txt
│   │   └── …
│   └── …
└── nametag_response_<doc_id>.conllu.json   # raw JSON reply from the NameTag API

[!TIP] If disk space is a concern you can safely delete TEMP/ once <OUTPUT_DIR>/UDP/ and <OUTPUT_DIR>/NE/ have been fully populated and step 4 has completed without errors. The paradata JSONs in <OUTPUT_DIR>/paradata/ and the processing.log are the only runtime records worth keeping long-term.

One-command pipeline run (`run_pipeline.py`)

# Full core run: api_1 → api_2 → api_3 → api_4
python3 run_pipeline.py

# Core run plus keyword extraction (CPU-only YAKE backend, default)
python3 run_pipeline.py --kw

# Keyword extraction with the GPU KeyBERT backend
python3 run_pipeline.py --kw --kw-method keybert

# Add the optional LLM semantic-enrichment stage (needs requirements_llm.txt)
python3 run_pipeline.py --kw --llm

# Run only a subset of the core stages (canonical order is always enforced)
python3 run_pipeline.py --stages udp nt

# Resume after an interruption: start from a chosen stage, skip every earlier one
python3 run_pipeline.py --start-from nt

# Skip individual stages (re-run only NER + stats, leave manifest/UDPipe as-is)
python3 run_pipeline.py --skip-manifest --skip-udp

# Clear stale .state_* checkpoint sidecars from PARADATA_DIR before running
python3 run_pipeline.py --clean-state

# Force execution: bypass missing dependency checks and ignore individual stage failures
python3 run_pipeline.py --kw --kw-method keybert --force

# Validate configuration and resolve the plan without running anything
python3 run_pipeline.py --dry-run

# Print the resolved config + stage plan as JSON (for wrappers / healthchecks)
python3 run_pipeline.py --print-config json

The runner reads the same config_api.txt 📎 that the shell stages source, so Python and Bash always agree on OUTPUT_DIR, PARADATA_DIR, and the input/output paths.

What the runner does

Resolves config from config_api.txt 📎 (with $VAR / ${VAR} expansion).
Runs each stage in order, spacing stage starts by ≥ 1.1 s so the 1-second-resolution paradata filenames (YYMMDD-HHmmss_nlp-enrich.json) never collide.
Collects the paradata JSON each stage writes, scoped to this run only (paradata files that already existed before the run are never merged).
Merges all per-stage records into one <PARADATA_DIR>/<runid>_nlp-enrich_pipeline-run.json via atrium_paradata.merge_run_paradata. The merged record accurately tracks document-level statistics across the sequential pipeline (recording true throughput without inflating input counts). The effective license of the merged record is re-derived from the union of every component used across the stages, so the most-restrictive rule holds end-to-end (a core run is CC BY-NC-SA 4.0; adding the YAKE backend escalates the share-alike/AGPL constraint, etc.).

Resume / checkpoint recovery

Pipeline-level starting points. To skip whole stages — not just completed documents — the runner accepts explicit entry points over the full stage order:

Flag	Effect
`--start-from <stage>`	Run from `<stage>` onward; every earlier stage is skipped.
`--skip-<stage>`	Skip one named stage, run the rest.
`--clean-state`	Sweep stale `.state_*.json` sidecars from `PARADATA_DIR` before running.

# UDPipe + NameTag already finished — resume at statistics, then keywords
python3 run_pipeline.py --kw --start-from stats

# Re-run only NER and statistics; keep the existing manifest and CoNLL-U
python3 run_pipeline.py --skip-manifest --skip-udp

Provenance for containers

ATRIUM_RUNNER_IMAGE="ghcr.io/ufal/atrium-nlp-enrich:v0.11.0" \
ATRIUM_RUNNER_REF="$(git rev-parse --short HEAD)" \
python3 run_pipeline.py --kw

Exit codes

Code	Meaning
`0`	All requested stages completed; nothing flagged.
`1`	A stage processed nothing despite having input and no resume, and `FAIL_ON_EMPTY=true` (the default).
`2`	A required stage script was not found.
`3`	A dependency preflight failed (e.g. `--kw-method keybert` without `keybert`/`sentence-transformers`, or `--llm` without the requirements_llm.txt 📎 stack).
`≠0`	A stage script itself exited non-zero (its code is propagated).

[!TIP] Using --force (-f) overrides exit codes 1, 3, and ≠0. It bypasses preflight dependency crashes and forces FAIL_ON_EMPTY=False, allowing the pipeline to continue attempting subsequent stages even if one stage crashes or processes zero files.

[!NOTE] The runner never re-implements stage logic: it shells out to the exact same api_1_manifest.sh … api_4_stats.sh, config_api.txt, and llm_run.py 📎 you can run by hand. Anything documented for those stages (resume behaviour, output flags, model registry, …) applies unchanged under the runner.

Acknowledgements 🙏

For support write to: lutsai.k@gmail.com responsible for this GitHub repository ²⁵ 🔗

Developed by UFAL ²⁶ 👥
Funded by ATRIUM ²⁷ 💰
Shared by ATRIUM ²⁷ & UFAL ²⁶ 🔗
Frameworks used:
- Lindat/CLARIAH-CZ NameTag 3 API ³ 🏷
- Lindat/CLARIAH-CZ UDPipe 2 API ² 🏷
- local KER (original lemma-frequency keyword extraction) ²⁸ 🏷
- YAKE (Yet Another Keyword Extractor, CPU statistical keyword extraction) ²⁹ 🏷
- KeyBERT (embedding-based keyword extraction, GPU-accelerated) ³⁰ 🏷
- UFAL flexiconv (format conversion to TEITOK XML) ⁵ 🏷

Similar repos

ufal/atrium-llm-enrich

Application of LLMs in context of the ATRIUM project for digitized documents enrichment

same org

UCREL/mosaico-usas-processing

Processing of the Mosaico dataset with USAS and CLAWS

impresso/impresso-named-entity-data-augmentation

Production of synthetic data to augment named entity ground truth

Digital-Scholarship-Oxford/crowdsourced-data-tools

Crowdsourced data tools

stonesalltheway1/Epstein-Pipeline

Open source document processing pipeline for the Epstein case files. Download OCR, extract…

116

adbar/trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction,…

6,798

michaelscho/lassberg

diazmarteth/NER

Named Entity Recognition pipeline with AI integration for Art and Architecture History.

Contributors

K4TEL

364 commits

dependabot[bot]

17 commits

Languages

Python

97.5%

Shell

1.5%

ufal/atrium-nlp-enrich

README

📦 ALTO XML Files Postprocessing Pipeline - NLP Enrichment of text

Table of contents

TEITOK XML — Unified Output Format

Why TEITOK XML?

TEITOK XML structure at a glance

⚙️ Setup

Workflow Stages

▶ Step 1: Prepare CSVs with texts from Page-Specific ALTOs

▶ Step 2: Extract NER and CONLL-U

Configuration ⚙️

Execution Pipeline

1. Generate Manifest

2. UDPipe Processing (Morphology & Syntax)

3. NameTag Processing (NER tags)

4. Generate Statistics

Inputs and Outputs

ALTO-to-TEITOK XML Generation and Coordinate Alignment

Fixing Bounding Box Alignments (Practical Guide)

Output Structure

EXTRA: Extract Keywords (KER / YAKE / KeyBERT)

Configuration Priority

Backends

Inputs and outputs

Score interpretation by backend

EXTRA: Converting Other Input Formats with flexiconv

What is flexiconv?

When to use flexiconv

How to use flexiconv

EXTRA: LLM Semantic Enrichment (Vocabulary Mapping)

⚙️ Configuration (llm_config.txt 📎)

🗂 Workflow

Reviewing the vocabulary

🖥 Model Registry

Single-GPU — BACKEND=transformers (or BACKEND=vllm)

MoE models — GGUF / llama.cpp fallback (any GPU, any VRAM)

MoE models — BACKEND=vllm (single GPU or multi-GPU)

Large models — BACKEND=vllm, TENSOR_PARALLEL_SIZE ≥ 2

📁 Inputs and Outputs

📊 Paradata Integration

EXTRA: REST API Service

Paradata Logs

<OUTPUT_DIR>/paradata/ — structured run logs 📂

<OUTPUT_DIR>/processing.log — human-readable runtime log 📄

TEMP/ — intermediate working files 📂

One-command pipeline run (run_pipeline.py)

What the runner does

Resume / checkpoint recovery

Provenance for containers

Exit codes

Acknowledgements 🙏

Footnotes

Contributors

Similar repos

Languages

ufal/atrium-nlp-enrich

README

📦 ALTO XML Files Postprocessing Pipeline - NLP Enrichment of text

Table of contents

TEITOK XML — Unified Output Format

Why TEITOK XML?

TEITOK XML structure at a glance

⚙️ Setup

Workflow Stages

▶ Step 1: Prepare CSVs with texts from Page-Specific ALTOs

▶ Step 2: Extract NER and CONLL-U

Configuration ⚙️

Execution Pipeline

1. Generate Manifest

2. UDPipe Processing (Morphology & Syntax)

3. NameTag Processing (NER tags)

4. Generate Statistics

Inputs and Outputs

ALTO-to-TEITOK XML Generation and Coordinate Alignment

Fixing Bounding Box Alignments (Practical Guide)

Output Structure

EXTRA: Extract Keywords (KER / YAKE / KeyBERT)

Configuration Priority

Backends

Single-GPU — `BACKEND=transformers` (or `BACKEND=vllm`)

MoE models — `BACKEND=vllm` (single GPU or multi-GPU)

Large models — `BACKEND=vllm`, `TENSOR_PARALLEL_SIZE ≥ 2`

`<OUTPUT_DIR>/paradata/` — structured run logs 📂

`<OUTPUT_DIR>/processing.log` — human-readable runtime log 📄

`TEMP/` — intermediate working files 📂

One-command pipeline run (`run_pipeline.py`)

Single-GPU — `BACKEND=transformers` (or `BACKEND=vllm`)

MoE models — `BACKEND=vllm` (single GPU or multi-GPU)

Large models — `BACKEND=vllm`, `TENSOR_PARALLEL_SIZE ≥ 2`

`<OUTPUT_DIR>/paradata/` — structured run logs 📂

`<OUTPUT_DIR>/processing.log` — human-readable runtime log 📄

`TEMP/` — intermediate working files 📂

One-command pipeline run (`run_pipeline.py`)