extraction-sandbox

Exploring entity extraction and resolution for knowledge graph construction — for agent memory and for document intelligence, which turn out to be genuinely different problems.

The first experiment is GLiNER2.5: a 194M-parameter encoder that does joint entity and relation extraction against a schema supplied at runtime, on a laptop CPU, with no API key.

The notebooks

No API key anywhere. Notebooks 03, 05, 06 and 07 use the claude CLI (already authenticated if you use Claude Code) through kgx.llm.ClaudeCLI, which caches every response to disk — a re-run is free and byte-identical. Notebook 09 uses no LLM at all, by design.

---

Quickstart

uv sync
uv run jupyter lab notebooks/01_gliner25_knowledge_graphs.ipynb

First run downloads ~400 MB (GLiNER2.5-base) plus ~90 MB (MiniLM, for entity resolution). Everything after that is local. No API keys anywhere in this repo.

import kgx

extractor = kgx.GlinerExtractor()                       # fastino/gliner2.5-base-v1
graphs    = extractor.extract_batch(docs, kgx.BUSINESS_NEWS)

resolver  = kgx.EntityResolver().learn_aliases(d["text"] for d in docs)
mentions  = [m for g in graphs for m in g.mentions]
kg        = kgx.build_graph(graphs, resolver.resolve(mentions), kgx.BUSINESS_NEWS)

print(kg.summary())
print(kgx.to_cypher(kg, min_support=2))

---

The pipeline

ontology  →  extract  →  [coref]  →  resolve  →  graph  →  [temporal]

Two ontologies ship: kgx.AGENT_MEMORY (13 node labels, 16 edge types) and kgx.BUSINESS_NEWS (13 / 15). Both are ordinary data — write your own in Python, JSON, or YAML.

---

The two use cases, and why they diverge

Same model, same joint decoding, different everything else.

---

Findings

Measured on this repo's corpora, not asserted. The notebook shows the working for each.

Joint decoding is the reason to use GLiNER2.5 over GLiNER2. Across both corpora, zero extracted edges violated the ontology's endpoint types. Extracting entities and relations separately gives you no such guarantee — you get a post-filter instead of a constraint.

Relation recall falls off a cliff past ~400 words, and lands on zero. A 518-word transcript decoded to 0 relations with feasible=True — indistinguishable from "no facts here". Entity recall over the same window was fine. Window the input: extract per episode, or extract_long(chunk_size=256..384), or JointIEConfig(max_len=512). kgx warns on this.

symmetric=True is broken in gliner2 2.0.0. It compiles to a constraint set that rejects every candidate edge — the relation silently returns nothing, with feasible=True. Use a directed relation plus inverse=, which works and emits the mirror edge tagged derived=True.

First-person substitution is worth ~500× on conversational text. Edges recovered about the user, over the same 5 sessions: 1 with raw turns → 210 with speaker labels → 539 with the full preprocessing stack. Not a model limitation; a preprocessing requirement of any coreference-free extractor.

Entity resolution: B-cubed F1 0.85 → 0.95, precision 1.00 throughout — all the movement is in recall. Embeddings buy the most F1, but mining aliases from the corpus text (Northwind Logistics Inc. (NASDAQ: NWL)) buys the most blocking recall, which is the harder ceiling: embeddings raise the score of pairs that are already candidates, alias mining creates candidates nothing else would propose. Complementary, not redundant.

Deciding what contradicts what needs world knowledge these models don't have. Embedding similarity does not separate genuine alternatives (npm/pnpm) from unrelated pairs (npm/Berlin) — the distributions overlap under every template tried. Asking the extractor to classify tools into categories fails too. What does work: declare replaces(tool → tool) in the ontology and let joint decoding find the switch the user announced in the text. Extracted at 0.99 confidence, with the sentence attached.

Errors move between stages wearing a disguise. A mistyped entity in extraction and an under-merge in resolution both surface as "the user changed their mind" in the temporal layer. A subject that flip-flops back to a value it already held is the tell. Carrying evidence on every edge is what makes them separable.

A prefix match is not an identity. Northwind / Northwind Logistics should merge; Apple / Apple Bank should not, and the string evidence is identical. Requiring context agreement splits them at no measured cost to F1 — the kind of rule that is invisible on a corpus without the trap and expensive on one with it.

---

Layout

src/kgx/            the library
  ontology.py       graph model as data; AGENT_MEMORY + BUSINESS_NEWS
  extract.py        GLiNER2.5 joint extraction + qualifier pass
  coref.py          conversation preprocessing (3 ablatable layers)
  resolve.py        entity resolution + incremental registry + B-cubed
  graph.py          canonical graph, evidence, matplotlib/pyvis/Cypher
  temporal.py       bi-temporal facts, supersession
  neo4j_io.py       idempotent Neo4j loading, schema introspection
  llm.py            cached claude-CLI backend + LLM extractor
  evaluate.py       triple scoring against gold
  baselines.py      spaCy closed-vocab baseline
  frameworks.py     LangChain / LlamaIndex / graphrag adapters
  gazetteer.py      controlled-vocabulary entity linking
  domains.py        travel / customer service / shopping ontologies
  data/             synthetic corpora with gold labels
notebooks/          01-08, see the table above
docs/LANDSCAPE.md   survey of the alternatives at every stage
output/             generated graphs, Cypher, CSVs (gitignored)

src/kgx/data/ ships 5 conversation sessions (80 turns) and 10 business-news documents (~2,300 words), both synthetic, both written with deliberate alias variation, planted contradictions, coreference stress, and modality traps — plus gold labels for entity resolution and a gold triple set. All companies, people, and events are fictional.

Neo4j

Notebook 02 needs a Neo4j 5.26+ instance. Nothing else — the embeddings are a local MiniLM and the two retrievers that genuinely need an LLM run against a deterministic stub (with a live path if ANTHROPIC_API_KEY is set).

scripts/neo4j_up.sh          # docker if available, a local tarball under .neo4j/ if not
uv run python scripts/neo4j_restore.py   # reload notebook 02's graph + the four indexes

Both are idempotent. neo4j_up.sh prefers Docker and falls back to a native install, so the notebooks keep working when Docker Desktop is down. neo4j_restore.py rebuilds the database from output/business_news_kg.json — the graph is fully reproducible, so a lost container costs a minute, not a re-extraction.

Override the connection with NEO4J_URI / NEO4J_USER / NEO4J_PASSWORD. Browse at http://localhost:7476.

More findings, from the Neo4j and comparison notebooks:

Dynamic labels work in 5.26 — you don't need APOC. MERGE (n:$($label)) takes the label as data. MERGE (n:$label) is a syntax error. APOC's apoc.merge.node(labels, ident, onCreate, onMatch) is a trap: pass properties only as onMatch and a first load writes nothing but the id — invisible on any database you have already loaded once.

CREATE INDEX ... IF NOT EXISTS is satisfied by an equivalent index under a different name. It succeeds, creates nothing, and the retriever fails several cells later with "No index with name … found".

Give every entity one shared :__Entity__ label. Cypher indexes are per label, so 13 ontology labels would mean 13 vector indexes. The shortcut create_fulltext_index(label="Company|Person") silently creates an index on one literal label named Company|Person that matches nothing, forever.

Entity lookup is a lexical task; rank it that way. HybridCypherRetriever with the default ranker returns Dresden for the query NWL. ranker="linear", alpha=0.2 fixes it.

Provenance filtering fixes multi-hop queries. A hallucinated subsidiary_of edge propagates into a two-hop ownership chain that never existed; filtering on r.support >= 2 inside the quantified path pattern prunes it before the path is built.

The "encoder cannot do implicit facts" claim was wrong (notebook 03). GLiNER2.5 recovered an implied relation, a cross-sentence syllogism and a bridged referent. The real boundary is about arguments: a relation can be inferred from context, an entity argument has to be anchored in the text. The same probe caught the encoder asserting a relation an explicitly negated sentence denies — which the LLM did not.

Splink beats the hand-rolled resolver, and so does an LLM tier (notebook 05). Both reach B³ F1 1.000 against the repo's 0.946, and both do it the same way: by fixing blocking recall (0.879 → 1.000), not by scoring better. Splink's own machinery contributes little here — EM training is worthless with one informative column, and its match probability is uncalibrated (optimal threshold 0.1, not 0.9).

Distillation worked and did not help (notebook 06). A full fine-tune learned the teacher's labelling function far better than its starting point (mention fidelity 0.453 → 0.641) and scored below its own zero-shot baseline on the benchmark. Seed variance inside one config (0.080 F1) exceeded the gap between the 74M and 194M models the experiment set out to close.

Coreference models lose to four regexes on chat (notebook 08). fastcoref added exactly zero gold facts over the deterministic layers; its small F1 edge came from emitting fewer spurious edges. All four engines fail on first person without a speaker prefix — and writing that prefix is what the rule layer does.

Frameworks mostly skip entity resolution (notebook 07). Bolting kgx.EntityResolver onto their output, with zero LLM calls, recovered ~96% of the gap between strict and alias-tolerant scoring.

Each NVL render() inlines an ~8.5 MB bundle, and from_neo4j copies every property — including your 384-float embeddings — into it. Strip them in Python; a map projection does not help.

"Precision 1.000" was a property of the corpus, not the resolver (notebook 09). Notebooks 01 and 05 both report perfect B-cubed precision on business news. Given an ecommerce corpus where Aurora 14 and Aurora 14 Pro are different products, the same code at the same threshold merges them — along with N600/N600X and Halcyon Buds/Halcyon Buds Pro. The distinguishing token is exactly the kind of short suffix normalisation is built to ignore.

Where a controlled vocabulary exists, stop computing similarity. A gazetteer links LHR to London Heathrow exactly, where Jaro-Winkler scores 0.45 and no threshold reaches it — and it yields a stable id that survives a rerun and a change of corpus, which clustering cannot. But validate any type constraint you put on it: matching the extractor's type against the vocabulary's kind cost 20 points of coverage and prevented zero errors, because both sides reasonably disagreed about whether an airport is a place.

Document-level judgement is the boundary of the no-LLM position. GLiNER2.5 classified support-ticket intent near chance and priority at chance — a near-constant predictor emitting high for seven of eight threads at 0.75–1.00 confidence. Intent is written down; severity is not, so a span model has nothing to key on. Of every task in notebook 09, that is the one worth escalating.

An ontology is a hypothesis about the text. replaces fires at 0.99 on agent memory ("I've switched to pnpm" — one clause, two named tools, an explicit verb) and never fires on support threads, where the same supersession is spread across a four-turn negotiation. Same relation, same model, different discourse shape.

Notes

• Requires Python ≥3.10 (this repo pins 3.12). sentencepiece and protobuf are required — GLiNER2.5 uses a DeBERTa-v3 SPM tokenizer and will fail confusingly without them.
• Load with AutoExtractor, never GLiNER2.from_pretrained — the latter is the legacy span loader and raises ArchitectureMismatchError on a 2.5 checkpoint.
• Extraction is not bit-reproducible; exact counts will shift slightly between runs.
• GLiNER2.5's published benchmarks are vendor-reported and unreplicated. Nothing here depends on them.