Knowledge Wiki

Transforms pre-chunked documents into an interconnected markdown wiki using GLiNER-RELEX — a single model that extracts entities and relations jointly, no pipeline required.

Output is a folder of plain .md files readable in Obsidian, any text editor, or directly by an LLM.

---

How it works

chunks (str / LangChain Documents)
        │
        ▼
  GLiNER-RELEX                  → entities + relations per chunk
        │
        ▼
  SBERT snippet ranker           → representative sentences per chunk (optional)
        │
        ▼
  Markdown writer
        │
        ├── entities/            one page per entity
        ├── relations/           one page per relation type
        ├── types/               one page per entity type
        ├── chunks/              one page per source chunk (provenance)
        └── INDEX.md             top-level navigation

Relations are stored in a consistent arrow format throughout every page:

[[cartels]] (group) -ally_of-> [[Kirk]] (person) | [chunk_0035](chunks/chunk_0035.md)

Entity pages use directional arrows to distinguish incoming vs outgoing:

-works_for-> [[Acme Corp]] (Organization)
<-manages-   [[Bob]] (Person)

---

Installation

pip install gliner sentence-transformers langchain-core tqdm numpy

The GLiNER-RELEX model weights (~500 MB) are downloaded from HuggingFace on first use.

---

Quickstart

from wiki import KnowledgeWiki

chunks = [
    "Alice works for Acme Corp in Singapore.",
    "Bob acquired Acme Corp last year for $2B.",
    "Alice and Bob were both founded by Carol.",
]

wiki = KnowledgeWiki(output_dir="wiki_output")
wiki.ingest(chunks)

Open wiki_output/INDEX.md in Obsidian or any markdown viewer to explore the result.

---

LangChain Documents

If your chunks come from a LangChain loader, pass them directly — source metadata is preserved and linked in each chunk page:

from langchain_community.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = TextLoader("report.txt")
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)
chunks = splitter.split_documents(loader.load())

wiki = KnowledgeWiki(output_dir="wiki_output")
wiki.ingest(chunks)

---

Configuration

KnowledgeWiki

Extractor

from wiki import Extractor, KnowledgeWiki

extractor = Extractor(
    model_name="knowledgator/gliner-relex-large-v1.0",
    entity_types=["Person", "Organization", "Location"],
    relation_types=["works_for", "located_in", "owns"],
    entity_threshold=0.4,    # minimum confidence to keep an entity
    relation_threshold=0.1,  # minimum confidence to keep a relation
    batch_size=4,
    use_cuda=True,
)

wiki = KnowledgeWiki(output_dir="wiki_output", extractor=extractor)
wiki.ingest(chunks)

SnippetRanker

from wiki import SnippetRanker, KnowledgeWiki

ranker = SnippetRanker(
    model_name="all-MiniLM-L6-v2",
    top_k=2,         # sentences to extract per chunk
    batch_size=64,   # sentence encoding batch size
    use_cuda=True,
)

wiki = KnowledgeWiki(output_dir="wiki_output", ranker=ranker)
wiki.ingest(chunks)

To disable snippet ranking entirely:

wiki = KnowledgeWiki(output_dir="wiki_output", ranker=False)

---

Output structure

wiki_output/
├── INDEX.md                     top-level links (entity types, relation types, entities A–Z)
├── entities/
│   ├── alice.md
│   ├── acme_corp.md
│   └── ...
├── relations/
│   ├── works_for.md
│   ├── acquired_by.md
│   └── ...
├── types/
│   ├── person.md
│   ├── organization.md
│   └── ...
└── chunks/
    ├── chunk_0000.md
    ├── chunk_0001.md
    └── ...

Entity page example (entities/alice.md)

# Alice
**Type:** Person

## Relationships
- -works_for-> [[Acme Corp]] (Organization)
- <-founded_by- [[Carol]] (Person)

## Representative Snippets
> Alice works for Acme Corp in Singapore.

## Source Chunks
- [chunk_0000](../chunks/chunk_0000.md)

## Backlinks
- [[Carol]]

Relation type page example (relations/works_for.md)

# Relationship: works_for
**Total instances:** 2

## Instances
- [[Alice]] (Person) -works_for-> [[Acme Corp]] (Organization) | [chunk_0000](../chunks/chunk_0000.md)
- [[Dave]] (Person) -works_for-> [[Acme Corp]] (Organization) | [chunk_0003](../chunks/chunk_0003.md)

## Backlinks (involved entities)
- [[Acme Corp]]
- [[Alice]]
- [[Dave]]

---

Tips

Thresholds — lower relation_threshold (e.g. 0.05) recalls more relations at the cost of noise. Raise entity_threshold (e.g. 0.6) to reduce spurious entity detection.

Batch size — larger batches are faster on GPU but use more VRAM. If you hit OOM errors, reduce batch_size to 4 or 2.

CPU-only — set use_cuda=False on both Extractor and SnippetRanker, or pass it to KnowledgeWiki and it will propagate automatically.

Re-ingesting — ingest() accumulates state. To start fresh, create a new KnowledgeWiki instance rather than calling ingest() twice on the same object.

Obsidian — open wiki_output/ as a vault. [[wiki links]] resolve automatically and the graph view shows the full entity-relation network.

---

Dependencies