---

🎯 TL;DR

---

Architecture

Two pipelines converge: one distills the law into geometry, the other teaches a language model to read that geometry.

German StVO legal text is parsed, translated via an LLM translator, and structurally aligned with a corresponding English-based knowledge graph, injecting explicit relational constraints into model reasoning. The TransE paragraph embeddings enter the LM pipeline as soft targets for the CLU objective — the only coupling between the two pipelines, and the mechanism by which legal structure reaches the model weights.

The total objective is a weighted sum over the unit simplex:

$$\mathcal{L}{total} = \lambda{NSP}\mathcal{L}{NSP} + \lambda{CLU}\mathcal{L}{CLU} + \lambda{CLS}\mathcal{L}_{CLS}, \qquad \sum\lambda = 1.0$$

The three objectives

CLU is the conceptual core. Because the target e⁺ is a 500-dimensional continuous vector rather than a discrete label, the gradient carries relational geometry — the model learns a continuous range of legal concepts instead of hard class boundaries.

Gated KG-aware attention

The CLS head does not rely on the self-contained encoder representation. It queries the whole paragraph-embedding library at decision time:

$$\alpha_j = \text{softmax}\left(\frac{\mathbf{z}_s^\top \mathbf{e}j}{\sqrt{d_h}}\right), \quad \mathbf{c}{KG} = \sum_j \alpha_j \mathbf{e}_j$$

$$\tilde{\mathbf{z}}s = g \odot \mathbf{c}{KG} + (1-g) \odot \mathbf{z}_s, \quad g = \sigma([\mathbf{z}s; \mathbf{c}{KG}]W_g)$$

Unlike retrieval-augmented generation, there is no discrete database lookup at inference — retrieval happens in latent space and is fully differentiable. The gate g can drive the KG contribution to zero when the retrieved context is uninformative, so the mechanism degrades gracefully rather than injecting noise. Implementation: multi_task_classes_STvO_mlm_clm_clu_cls.py.

---

Results

Main — synthetic StVO QA

Multi-task pre-training (PT) is zero-shot; FT adds a few downstream samples. Gains are largely decoupled from base performance, which is the evidence that the objectives — not a strong starting point — are doing the work.

bert-base-eurlex climbs from the weakest base (0.338) to 0.862. Counterintuitively, general-purpose google-bert-base gains more (+0.205) than the broad multi-field legal-bert-base (+0.114) — we attribute this to domain-specific interference, where many legal-English tasks conflict with the StVO domain, while single-subfield specialists (echr, eurlex) align cleanly.

Cross-lingual transfer (RQ1)

NSP and CLU are trained on non-translated German StVO; only the CLS head sees English. Any zero-shot gain is therefore German-learned structure surfacing in English.

bert-base-eurlex barely improves with further fine-tuning (+0.032) — the German NSP+CLU signal was already sufficient. Models that fail zero-shot recover fully once fine-tuned, all converging to a narrow 0.862–0.878 band.

Out-of-domain legal transfer (RQ3) — LexGLUE

Trained exclusively on German traffic law, the models set new SOTA on four metrics across four English legal tasks:

† new SOTA over the published baselines of Chalkidis et al. (2021). The models show negative transfer on SCOTUS, EUR-LEX, and LEDGAR — reported in full rather than omitted.

Ablation — which objective matters?

Average m-F1 drop when each objective is removed, across all nine models:

The full NSP+CLU+CLS configuration wins for 7 of 9 models. CLS dominates because it aligns directly with the downstream task. CLU's aggregate effect is not statistically significant — it helps top performers (bert-base-echr: 0.587 → 0.864) but interferes in bert-base-contracts and google-bert-base, where random negative sampling can pair semantically related paragraphs as negatives. Task synergy is architecture-dependent, and we report that honestly.

---

Datasets

Synthetic StVO QA is published separately — the corpus, the 11 individual generation runs with their manifests, and the full generation/validation/evaluation pipeline live at stvo-legal-qa, together with a dataset card documenting the record schema, the paragraph-disjoint split, and known label noise.

Synthetic QA generation

legal_qa_generation_multi_lingual.py prompts Llama-2-7b-chat, Mistral-7B-Instruct-v0.2 and Chocolatine-14B to produce, for each StVO sentence, a source-grounded correct answer and a plausible but contradictory incorrect one.

A six-stage validation pipeline then filters them — structural and field constraints, rejection of non-reasoning yes/no questions, domain-terminology verification (Vorfahrt, right of way), lexical-overlap removal of trivial paraphrases, and placeholder detection. Only pairs scoring Q ≥ 0.6 on a composite metric survive. Validation and analysis tooling lives in data/synthetic-data-eval/.

Critically, 17 % of paragraph identifiers are reserved exclusively for testing, so the model cannot exploit content memorisation — the split is by paragraph, not by pair.

Example — § 8 Right of way Source: "At intersections without traffic signs or lights, vehicles coming from the right have right-of-way." ✅ Correct: Q: "Which vehicle has priority at an unmarked intersection?" → A: "…vehicles approaching from the right have right-of-way." ❌ Incorrect: Q: "Who must yield at an intersection without signals?" → A: "The vehicle traveling at lower speed must always yield to faster-moving traffic." (contradicts the right-hand priority rule)

---

Resources

Everything the pipeline consumes, and where it comes from. Nothing here is redistributed by this repository except the KG embeddings under legal-KGE/.

Primary source

The parser scrapes this URL directly, pulling the ~500 referenced Zeichen images alongside the text. As an amtliches Werk under § 5 UrhG the statute carries no copyright, but it is amended over time — a re-scrape today yields a slightly different graph than the 554-node/945-edge snapshot reported here.

Derived resources — published separately

stvo-legal-qa carries the corpus this project trains and evaluates its classification head on, so the numbers in Results are reproducible without rerunning generation: clone it and point --dataset_path at data/stvo/qa_train.jsonl / qa_test.jsonl. Its paragraph-disjoint split is the same one used here — 49 training §§, 10 held-out §§, zero overlap.

Models

Gated checkpoints (Llama, Mistral) need a Hugging Face token, read from HF_TOKEN.

Evaluation benchmarks

---

Repository layout

├── legal-parser/                 Stage 1 — legal text → knowledge graph (published standalone)
│   ├── main_content_extractor.py     Structured content extraction
│   ├── table_content_extractor.py    Tables and traffic-sign figures
│   ├── citator.py                    Cross-reference / citation resolution
│   ├── knowledge_graph_constructor.py
│   └── kg_class.py
│
├── legal-KGE/                    Stage 2 — knowledge graph embeddings
│   ├── train_legal_kge.py            TransE · DistMult · RotatE · ComplEx
│   ├── train_transE.py               Standalone TransE trainer
│   ├── extract_paragraph_embeddings.py
│   ├── compare_kge_models.py
│   └── embeddings/                   Pretrained checkpoints (committed)
│
├── legal_qa_generation_multi_lingual.py    Stage 3 — synthetic QA generation
├── data/synthetic-data-eval/               QA validation & analysis suite
│   ├── evaluation_suite.py · llm_evaluator.py · report.py
│   ├── validate_qa_dataset.py            Six-stage quality filter
│   └── paper_visualize.py · tables.py    Paper figures and tables
│
├── finetune_multi_task_STvO_mlm_clm_clu.py      Stage 4 — training loop
├── multi_task_classes_STvO_mlm_clm_clu_cls.py   Model, datasets, KG attention
├── segmenter.py                                 Structure-aware segmentation
│
├── eval.py                       Stage 5 — evaluation entry point
├── driving_examiner.py               Prompting, retrieval, scoring
├── driving_license_data_loader.py    DE / IE / AT exam loaders
├── plotter.py                        Figures
├── results/stats_table.py            Result aggregation
│
├── benchmark-lexglue/            LexGLUE modifications (see below)
├── slurms/                       SLURM job scripts (HPC)
└── tmux/                         tmux runners (single-node multi-GPU)

Data and results — data/, results/, plots/, and stats/ hold large artifacts and are excluded from version control; the scripts inside them are tracked. Pretrained KG embeddings under legal-KGE/embeddings/ are committed, so evaluation reproduces without re-running stages 1–2.

LexGLUE — the RQ3 experiments run on coastalcph/lex-glue, which is a separate upstream repository. Rather than vendoring it, benchmark-lexglue/ carries our modifications as a patch. Clone upstream, then git apply ../benchmark-lexglue/stvo-lexglue-modifications.patch.

---

Getting started

git clone https://github.com/ibrahimssd/stvo-llm-driver.git
cd stvo-llm-driver

python3 -m venv envs/driving-env
source envs/driving-env/bin/activate
bash tmux/install.sh

install.sh pins PyTorch 2.6.0 / CUDA 12.4 and installs transformers, datasets, accelerate, peft, evaluate, scikit-learn, wandb, and the plotting stack. All experiments ran on a single NVIDIA H100 80 GB, seed 42.

KG Pipeline

1 · Parse the statute into a knowledge graph — prerequisite. This step is not performed in this repository. The pattern-based parser is published standalone as stvo-legal-knowledge-graph-parser (ESSV 2025), and everything below consumes its output. It exploits the semi-structured hierarchy of the StVO — categories, paragraphs, Absätze, Sätze, tables, sections, Zeichen — and resolves the statute's own cross-references into typed edges, using a predefined ontology rather than an LLM-inferred one.

git clone https://github.com/ibrahimssd/stvo-legal-knowledge-graph-parser.git
cd stvo-legal-knowledge-graph-parser
pip install -r requirements.txt

bash scripts/1_extract_entities.sh     # scrape gesetze-im-internet.de, tag the tree
bash scripts/2_translate.sh            # DE → EN, m2m100_418M (optional)
bash scripts/3_construct_kg.sh         # build the integrated graph

Step 2 is the DE → EN hop in Figure 1; --model_name Helsinki-NLP/opus-mt-de-en produces the opus-mt-de-en_* variants used in the translation ablation. Step 3 needs neither the scrape nor the translation — it builds from the parser repo's examples/ in seconds. Output: 554 nodes, 945 edges, 5 relations.

Copy the resulting files into data/stvo/. Four of them feed this project directly, one per downstream consumer:

The _graph_raw.txt node dump is additionally available for plain MLM pre-training. Filenames in data/stvo/ carry the translation model as a prefix (m2m100_418M_translated_…, opus-mt-de-en_translated_…), which is how the translation ablation selects between them.

2 · Train the KG embeddings — TransE, producing the 500-d paragraph embeddings eₚ used as CLU soft targets.

python legal-KGE/train_legal_kge.py \
    --model_type transe \
    --data_path data/stvo/stvo_triples.txt \
    --embedding_dim 500 \
    --epochs 50 \
    --batch_size 256 \
    --learning_rate 1e-3 \
    --margin 1.0 \
    --save_paragraph_embeddings \
    --output_dir legal-KGE/embeddings/transe/STvO_de

Swap --model_type for distmult, rotate, or complex to reproduce the preliminary KGE comparison (legal-KGE/compare_kge_models.py); TransE was the most robust on downstream tasks.

LM Pipeline

3 · Generate and validate the QA corpus.

python legal_qa_generation_multi_lingual.py \
    --input_file data/stvo/structured_stvo.json \
    --output_file data/synthetic-data/stvo_qa_de.json \
    --language de --domain_type traffic \
    --pairs_per_sentence 2 --min_quality_score 0.6

python data/synthetic-data/stvo/validate_qa_dataset.py   # six-stage quality filter
bash data/synthetic-data-eval/run_eval_pipline.sh        # corpus analysis + report

4 · KG-guided multi-task pre-training — the base encoder internalises legal structure via NSP + CLU + CLS.

python finetune_multi_task_STvO_mlm_clm_clu.py \
    --base_model_name nlpaueb/bert-base-uncased-echr \
    --modeling_type masked \
    --legal_text STVO \
    --clm_data_path data/stvo/clm.json \
    --nsp_data_path data/stvo/nsp.json \
    --clu_data_path data/stvo/clu.json \
    --cls_data_path data/stvo/cls.json \
    --paragraph_embeddings_file legal-KGE/embeddings/transe/m2m100_418M_STvO/transe_paragraph_embeddings_best.json \
    --clm_nsp_lambdas 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 \
    --cls_lambdas 0.1 \
    --num_epochs 20 --batch_size_nsp 16 --max_segments 10 \
    --learning_rate 5e-5 --gradient_accumulation_steps 2

λ_CLU is derived as 1 − (λ_NSP + λ_CLS), giving 36 valid configurations per model.

5 · Evaluate — zero-shot on the StVO QA test split, the Irish and Austrian exams, and (via benchmark-lexglue/) LexGLUE.

python eval.py \
    --data_path data/test-sets/irish-driving-license.json \
    --data_name irish \
    --baseline_model nlpaueb/bert-base-uncased-echr \
    --model_checkpoint_dir  <checkpoint-dir> \
    --model_checkpoint      m2m100_418M_clm_nsp_0.4_clu_0.5_cls_0.1.pt \
    --paragraph_embeddings_file legal-KGE/embeddings/transe/m2m100_418M_STvO/transe_paragraph_embeddings_best.json \
    --modeling_type masked --task_type cls --num_cls_labels 2 \
    --decoding combine --similarity_threshold 0.6 \
    --skip_images --skip_category direct_answer \
    --results_file results/irish-echr.csv

Add --eval_base for the untuned baseline. Full sweeps: cd tmux && bash stvo_train_multi_task.sh or cd slurms && sbatch stvo_train_multi_task_ultimate.sm.

---

Hyperparameters

---

Research questions

• RQ1 — Does the semi-structured nature of StVO enable cross-lingual transfer of QA legal reasoning? → Yes; five models show positive zero-shot gains up to +0.499.
• RQ2 — What are the limits of cross-domain generalisation to foreign driving regulations? → All models improve on Austrian (+0.041 to +0.158); Irish gains are smaller (+0.002 to +0.082). Linguistic distance is a stronger barrier than regulatory distance.
• RQ3 — Does StVO knowledge transfer to other legal tasks? → New SOTA on 4 of 7 LexGLUE metrics, with negative transfer on 3.

---

Limitations

• Domain and language scope — validated only on German traffic regulations. Generalisation to other legal systems and languages remains empirically open.
• Synthetic benchmark — the 12,554 QA pairs are LLM-generated and may not reflect authentic examination distributions, particularly for ambiguity and multi-hop reasoning. Because training and evaluation pairs share generators, gains are evidence on synthetic legal QA, not authentic exams.
• CLU negative sampling — random negatives can pair semantically related paragraphs, causing interference for out-of-domain bases such as bert-base-contracts. Hard-negative sampling is future work.
• KG dependence — the graph is parsed from an LLM-translated English StVO; translation and embedding noise are not quantified. CLS attention scales over the full paragraph set, tractable for 59 paragraphs but untested at larger graph sizes.
• Capacity and linguistic compatibility — legal-bert-small plateaus (capacity bottleneck) and Legal-heBERT degrades monotonically (Hebrew↔English incompatibility), showing severe precision-recall imbalance (P = 1.000, R = 0.096).

Ethical considerations

These are research prototypes, not legal advice. Outputs must not substitute for a qualified legal professional, and deployment in safety-critical settings such as automated driver assessment without human oversight is strongly discouraged. The corpora are LLM-generated and quality-filtered; filtering does not guarantee correctness, and no held-out subset was validated by human legal experts. Base encoders pre-trained on corpora such as EUR-LEX may encode biases that continued StVO pre-training does not remove.

---

Citation

@inproceedings{siddig2026stvo,
  title     = {Enhancing Legal Reasoning in Pre-trained Language Models via
               Knowledge Graph-Guided Multi-Task Pre-training},
  author    = {Siddig, Ibrahim and Georges, Munir},
  booktitle = {Proceedings of the Natural Legal Language Processing Workshop 2026 (NLLP)},
  year      = {2026}
}

The pattern-based parser behind the knowledge graph is published separately as stvo-legal-knowledge-graph-parser:

@inproceedings{siddig2025parsing,
  title     = {Pattern-based Parsing of German Traffic Regulations (StVO)
               for Legal Knowledge Graph Construction (KGC)},
  author    = {Siddig, Ibrahim and Tugeev, Sviatoslav and Georges, Munir},
  booktitle = {ESSV},
  year      = {2025}
}

---