Harness vs Model

Can a locally-run general LLM (Gemma 4 12B) replace a purpose-built 200M specialist (GLiNER-Relex) for knowledge-graph extraction? I ran the test, and the answer turned out to depend on a single default setting far more than on the model itself.

The question

If you build with AI agents, you hit the context problem: agents lose the thread every time they hand work to each other. One way to give them shared context is a knowledge graph. But a knowledge graph has to be built, and that means pulling entities and the relationships between them out of plain text first.

I run everything locally, and I did not want to load a second model just for that extraction step. So the question was simple: can the LLM I already run (Gemma 4 12B) do the extraction itself, or do I still need a dedicated specialist?

To answer it, I benchmarked Gemma 4 12B against GLiNER-Relex, a 200M model built only for this, on CoNLL04 (joint named-entity and relation extraction).

TL;DR

• Run the default way, Gemma tied the tiny specialist on relations while running about 200x slower and failing on roughly 1 input in 9.
• Gemma 4 ships with a "thinking" (step-by-step reasoning) mode on by default. Turning it off raised relation F1 from 0.33 to 0.51, cut failures from 11% to 0%, and dropped latency from about 139s to about 10s per sentence.
• With thinking off, Gemma beat the specialist on quality. The specialist is still far cheaper (0.6s per sentence on CPU) and cannot produce a broken answer by design, because it points at words already in the sentence instead of generating text.
• The result came down to the harness (the system around the model) more than the model itself.

Results

CoNLL04 test set, 288 sentences, strict micro-F1.

Notes:

• The vague "Other" entity type scores about 0.06 to 0.09 F1 for both models and drags the overall entity number down. Strip it out and both look much stronger. See outputs/per_type_f1.csv.
• The thinking-off gain is not only from fixing the failures. On the sentences where thinking-on did return an answer, thinking-off still scored higher, so the reasoning was also degrading otherwise-valid answers.
• Full charts for every finding are in analysis.ipynb.

Setup

Requirements:

• Python 3.12
• Ollama with gemma4:12b pulled (the 4-bit build runs in about 16GB of RAM)
• GLiNER-Relex downloads automatically from Hugging Face on first run

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
ollama pull gemma4:12b

How to run

cd src

# run both models over the test set (thinking on by default)
python run_experiment.py

# run Gemma with thinking off
python run_experiment.py --thinking=False

# quick smoke test on a subset
python run_experiment.py --thinking=False --limit=15

# build the result tables from saved predictions
python make_report.py

The runner writes to outputs/predictions.csv incrementally and is resumable. Stop it and rerun, and it continues where it left off. Thinking-on and thinking-off results are stored under separate model labels, so a second run appends rather than overwrites. make_report.py reads those predictions and writes results.csv, per_type_f1.csv, and summary_table.md.

Methodology

• Both models see the same 288 CoNLL04 test sentences and return a list of entities and a list of relations.
• Before scoring, both models' labels are mapped to one shared set (one model says "location" where the dataset says "Loc").
• Scoring is strict micro-F1. An entity is correct only if its text and its type both match the gold answer. A relation is correct only if the subject, the relation, and the object all match. No partial credit, no semantic matching.
• Nothing is tuned on the test set. Thresholds are fixed at the model authors' defaults, and the same minimal, symmetric post-processing is applied to both models.
• Gemma uses schema-constrained JSON output so its labels stay inside the allowed set.
• Alongside F1, the pipeline logs what F1 ignores but a real system cares about: parse-failure rate, hallucinated-entity count, and per-sentence latency.

Problems encountered

The most interesting part was Gemma's 11% failure rate, and the fact that the cause was not what it first looked like.

Reading the raw outputs and Ollama's token-level timing, I traced the failures to three modes:

1. Runaway reasoning (25 of 32 failures). With thinking on, on hard sentences the model reasoned in circles until it filled the entire context window (about 4096 tokens) and never wrote an answer. One reproduced case generated 3,886 tokens with done_reason=length and an empty answer. Disabling thinking fixed it (8 to 10 seconds, clean answer), and it was also most of the 138s average latency.
2. Repetition loops (4 of 32). At temperature 0 (greedy decoding), on a low-confidence word the model would emit a stray digit and then loop on it (100000...) until it was cut off.
3. Stray special tokens (3 of 32). A valid answer with an internal marker (<|tool_response>) glued onto the end, which broke strict JSON parsing. A defensive parser that extracts the outermost {...} recovers these.

The diagnosis was not obvious, and ruling out two plausible causes with data was part of the work:

• A token cap (num_predict) truncating answers. Ruled out, because the failures were the longest runs producing empty or looping output, not short truncations of good answers.
• The JSON-schema grammar causing the roughly 10x slowdown. Ruled out by a clean attribution (thinking on/off crossed with schema on/off), which showed the grammar adds only about 20% and that the real cost was hidden thinking tokens, which Ollama's eval_count does not report.

Repository layout

src/
  schema.py          canonical data model (Entity, Relation, Example, Prediction) and shared label maps
  load_data.py       CoNLL04Loader: loads and reconstructs gold entities and relations
  run_gliner.py      GLiNERExtractor: the 200M specialist
  run_gemma.py       GemmaExtractor: local Gemma via Ollama, schema-constrained output, thinking toggle, defensive parsing
  run_experiment.py  resumable runner, writes predictions.csv, --thinking and --limit flags
  evaluate.py        Evaluator: strict micro-F1, per-type F1, and the diagnostics
  make_report.py     writes results.csv, per_type_f1.csv, summary_table.md
outputs/             generated results
analysis.ipynb       the full visual analysis (a chart for every finding)

A note on the metric

I report strict micro-F1: pool every entity and relation type into one score, with exact matching. It is the standard for this benchmark and it keeps the comparison honest, since there is no tunable leniency to accidentally inflate a score. It also makes the numbers conservative ("the United States" vs "United States" counts as a miss), but the same strict rule is applied to both models, so the comparison stays fair.

Takeaway

The model was capable the whole time. What decided the outcome was the harness around it: the prompt, the output format, the decoding settings, and the parsing. A single default flag flipped Gemma from a slow, unreliable tie into the most accurate model in the test. For a local setup, how you run a model can matter as much as which model you pick.