---

📦 Keyword extraction — statistical and vocabulary-controlled

Keywords of archival documents, of two kinds kept apart — and every keyword says which method produced it, with its score and rank:

• statistical keywords — KeyBERT (the default), YAKE (selectable) and the legacy KER, from the document or from each of its pages;
• controlled keywords — per line, the term of the AMČR and TEATER vocabularies a language model chooses for it (with the AMČR/TEATER records behind the term), the Czech and English keywords found in the line, and the page and line it came from, plus entity links to AMČR and AAT.

One service, POST /extract_keywords, takes the ATRIUM document record after nlp-enrich and answers with the statistical keywords of the document and of its pages, and the record with its controlled keywords written in (the enrichment block). This repository is where both kinds live: it was assembled on 1 October 2026 from the keyword extraction of atrium-nlp-enrich (keywords.py) and of atrium-llm-enrich (the LLM engine and the vocabulary code), after the meeting of 30 September (atrium-keyword-extract#1; atrium-project#72). nlp-enrich keeps only the LINDAT calls (UDPipe, NameTag) and what is built from them.

[!NOTE] State of this branch. Both kinds run in the service. The controlled kind calls a language model in an inference service — OpenRouter, or a local Ollama — configured by LLM_BACKEND; a deployment without one answers kind=controlled with 501 and reports it skipped under kind=both. Its quality is not evaluated yet: the evaluation rubric (D1) of atrium-keyword-extract#2 was decided on 8 October 2026, but the annotated corpus and the scorer (D2) do not exist yet, so treat its output as research output. Since v1.2.0-beta the statistical keywords are also written into the record, as its keywords block (atrium-project#73).

---

📖 Table of Contents

• The service
• Setup
• The batch CLI: keywords.py
• The controlled kind and the vocabulary
• Licences
• Docker
• Contributing and contacts

---

The service

service/README.md is the reference: the endpoints, the options, the response, the limits, the errors and the OpenAPI contract. In short:

python -m service.api                  # PORT / HOST; default 0.0.0.0:8000

curl -s -X POST localhost:8000/extract_keywords \
  -F "document_json=@CTX000000001.document.json;type=application/json" \
  -F kind=statistical -F method=keybert -F num_keywords=10

# both kinds, with the controlled one's backend configured
OPENROUTER_API_KEY=sk-... OPENROUTER_MODEL=openai/gpt-4o-mini python -m service.api
curl -s -X POST localhost:8000/extract_keywords \
  -F "document_json=@CTX000000001.document.json;type=application/json" -F kind=both

The record is read: its lines[].text per page (lines labelled Trash, Garbage, Inverted or Empty are left out), and for the legacy method its lines[].lemma and lines[].upos, which nlp-enrich writes. In the pipeline it comes after nlp-enrich (and after digital-convert on the born-digital route), and its keywords can be projected into the TEITOK header by nlp-enrich's /project_record.

Scores are compared only within one method.

Setup

git clone https://github.com/ufal/atrium-keyword-extract.git
cd atrium-keyword-extract
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt                      # KeyBERT, YAKE, the shared contract
pip install -r service/requirements.txt              # + the service (fastapi, uvicorn)

KeyBERT downloads its embedding model (paraphrase-multilingual-MiniLM-L12-v2) from the Hugging Face Hub on first use; set HF_HOME to keep the cache. A GPU is used when torch finds one. The environment of the service is .env.example.

---

The batch CLI: keywords.py

[!NOTE] The command-line form of the statistical kind, for a whole collection at once. It reads the CoNLL-U files nlp-enrich writes (or its TEITOK files) and writes CSV tables; the service reads the document record instead. Both run the same three methods.

Extract keywords 🔎 from your documents by running keywords.py on a directory of CoNLL-U files.

Configuration Priority

The keyword extraction script uses a three-tier configuration hierarchy (from highest to lowest priority):

1. Command-line flags (e.g., -m yake, -w 3) always override everything else.
2. kw_config.txt (the [DEFAULTS] section) is read automatically if placed next to the script.
3. Hardcoded fallbacks are used if no config file or flags are provided.

This means if you configure your settings in kw_config.txt, you can simply run:

python3 keywords.py

Backends

You can override any kw_config.txt setting via the command line:

python3 keywords.py -i <input_dir> -m <method> -l <lang> -w <integer> \
                    -n <integer> -d <output_dir> -o <output_file>.csv

All available flags:

Examples:

YAKE — Czech, up to 3-word phrases, 20 keywords per document

python3 keywords.py -i OUTPUT_DIR/UDP -m yake -l cs -w 3 -n 20 \
        -o keywords_summary.csv -d KW_PER_DOC

KeyBERT — multilingual model, GPU-accelerated (the default)

python3 keywords.py -i OUTPUT_DIR/UDP -m keybert -w 3 -n 20 \
        --keybert-model paraphrase-multilingual-MiniLM-L12-v2 \
        -o keywords_summary.csv -d KW_PER_DOC

Legacy KER — (English/Czech) original ATRIUM lemma-frequency approach, no extra dependencies

python3 keywords.py -i OUTPUT_DIR/UDP -m legacy -n 20 \
        -o keywords_summary.csv -d KW_PER_DOC

[!WARNING] For KeyBERT with a GPU, the script automatically forces --workers 1 to prevent competing CUDA context initialisation across subprocesses. On CPU, any worker count is safe.

Inputs and outputs

• Input: Directory of per-document CoNLL-U files (nlp-enrich's UDP/), or *.teitok.xml files.
• Output 1: Master table with keywords per document (e.g., keywords_summary.csv).
• Output 2: Per-document CSV files (e.g., KW_PER_DOC/).

KW_PER_DOC/
├── <docname1>_keywords.csv
├── <docname2>_keywords.csv
└── ...

Each per-document file contains two columns — keyword and score — sorted by score in descending order. The master summary uses the same column structure as the original pipeline (document_id, kw-1, score-1, kw-2, score-2, …).

Score interpretation by backend

legacy — raw lemma count; higher = more frequent in the document. Examples in directory: KW_PER_DOC_L 📂 and summary file kw_summary_l.csv 📎.

yake — normalised inverse YAKE score, [0, 1] per document. Examples in directory: KW_PER_DOC_Y 📂 and summary file kw_summary_y.csv 📎.

keybert — cosine similarity to document centroid, [0, 1]. Examples in directory: KW_PER_DOC_KB 📂 and summary file kw_summary_kb.csv 📎.

---

The controlled kind and the vocabulary

The controlled kind maps each line onto the AMČR keyword lists and the TEATER thesaurus with a language model whose answer is constrained to the vocabulary's terms (a JSON schema whose category is an enum of its 4711 terms and the meta-text answer Nerelevantní (meta-text)). One prompt serves every path: prompts/system_prompt.txt, whose blocks the PROMPT_* flags of llm_config.txt select — prompts/RUNBOOK.md explains each — followed by the vocabulary and the examples; python3 prompt_template.py --preview prints it. Each answer carries the AMČR/TEATER records behind the chosen term (teater_category_ids), and a homonym the vocabulary build qualified (zámek (sídlo elity)) comes back as its plain label.

The first two share llm_client_shared.py, which needs no GPU stack (requirements_remote.txt); the third is the bake-off path of atrium-keyword-extract#2 (llm_utils.py, requirements_llm.txt, the llm Docker target).

pip install -r requirements_remote.txt
export OPENROUTER_API_KEY=sk-...
python openrouter_client.py --input CTX000000001.document.json --model openai/gpt-4o-mini \
  --document-json-out CTX000000001.kw.document.json
python ollama_client.py --input data_samples/DOC_LINE_CATEG --model qwen2.5:14b

The research run on a GPU node:

pip install -r requirements.txt -r requirements_llm.txt   # vLLM brings its own torch pin
$EDITOR llm_config.txt     # MODEL_KEY; INPUT_DIR (line CSVs or *.teitok.xml), OUTPUT_DIR, PARADATA_DIR
python llm_run.py --download   # where huggingface.co is reachable: MODEL_KEY into the HF cache
python llm_run.py          # or: python llm_run.py <another config>

A GPU node without internet access (HF_HUB_OFFLINE=1) loads a model only from the Hugging Face cache, so fetch it first on a machine that has access, into the same HF_HOME (--download KEY ... takes several keys). It fetches what a run reads (configs, tokenizer, weights; not README or LICENSE), skips files already cached (huggingface_hub then reports 0 B), and ends with what the cache holds. A run whose model files are not all cached stops before loading and names what is missing.

It writes <OUTPUT_DIR>_<model>/<doc_id>_enriched.json per document (and <doc_id>_enriched.abort.json after ten consecutive errors), skips a document whose output already exists, so a stopped job resumes (delete an aborted document's two files to redo it), and writes a paradata record naming the vocabulary build and the sha256 of the prompt it used. The whole prompt is about 58k tokens: a model served with a smaller window, including the large vLLM models at their default MAX_MODEL_LEN of 16384, sees only a prefix of the vocabulary, and the paradata records how many terms were cut. vLLM computes the prompt's shared part once and reuses it (prefix caching), and so does BACKEND=transformers with REUSE_PROMPT_CACHE=true; without it every line re-reads the whole prompt. PREFILL_CHUNK_SIZE keeps the transformers prefill within a 24 GB card. Hybrid models (Qwen3.5, Qwen3.6) need flash-linear-attention, which requirements_llm.txt lists: without it their linear-attention layers run on a reference kernel that is far slower and does not fit. The llm image carries no sample inputs, and whatever it writes outside /data goes with the container, so point INPUT_DIR, OUTPUT_DIR and PARADATA_DIR under /data in the llm_config.txt the compose file mounts.

The vocabulary:

• vocab_sources.py, vocab_build.py, vocab_review.py, vocab_manager.py — harvest the AMČR and TEATER vocabularies, build the nested union, review it; the built files are in data_samples/vocab (see its RUNBOOK). The vocabularies are CC0; the build is to be published as a versioned release asset for the translator and the end-to-end test.
• corpus_review.py — the vocabulary-gap review used for the TEATER experiment.

The LLM engine and the vocabulary code existed twice, in nlp-enrich and in llm-enrich, and partly diverged. This repository holds the one copy: nlp-enrich's came with the repository, and llm-enrich's (atrium-digital-convert 31534d5) was merged into it on 2026-10-07 — its remote clients, its service path and its later engine fixes.

Licences

The code is MIT. The licence of a run's output is computed from the components it used, as declared in para_config.txt, and the most restrictive one wins:

The language model of the controlled kind has no row: its terms depend on the model a deployment chooses (OPENROUTER_MODEL, OLLAMA_MODEL, MODEL_KEY), whose id every run records in its paradata. The source document's own licence applies to its text.

Docker

docker compose --profile api up                       # the service on :8000
docker compose run --rm kw -i /data/UDP -m yake       # the batch CLI
docker compose -f docker-compose.yaml -f docker-compose.gpu.yaml --profile llm run --rm kw-llm

The images of atrium-nlp-enrich (its -llm target) and atrium-llm-enrich that carried these stages before 1 October 2026 stay published for consumers pinned to them and receive no new tags.

Contributing and contacts

See CONTRIBUTING.md. For support write to lutsai.k@gmail.com, responsible for this repository.

• Developed by: UFAL ¹
• Funded by: ATRIUM ²

©️ 2026 UFAL & ATRIUM

Footnotes

1. https://ufal.mff.cuni.cz/home-page ↩

2. https://atrium-research.eu/ ↩