convograph

A pipeline for turning multi-party spoken conversation into a temporal knowledge graph and an agent memory system (AMS), with an optional path to real-time visual output (graphic recording).

_{Where we are heading. Four people talk. Diarization turns the audio into a speaker-tagged transcript; a temporal knowledge graph accumulates what was said and what has since been revised; a visual stage renders it onto a canvas that keeps updating as the meeting goes on — a graphic recording, produced live.
This is the target picture, not the current state: modules 1 (Sortformer diarization + multitalker transcription) and 2 are built, module 3 is in progress. FLUX.1-schnell and Excalidraw remain candidates under evaluation, not decisions.}

Setup

Everything heavy is hosted. You need no GPU, and you do not need to remember any SSH incantation.

1. ~/.ssh/config

Two services run on unicorn (DFKI's shared DSA box) and bind to 127.0.0.1, so they are reached through an SSH tunnel. Add this with your own DFKI username, and be on the DFKI VPN:

Host unicorn
  HostName serv-7101.kl.dfki.de
  User <your-dfki-username>
  IdentityFile ~/.ssh/id_ed25519

2. Secrets

Copy the git-ignored .env into modules/kg-agent-memory/ (ask Faris). It holds the AuraDB connection and SAIA_API_KEY. Never commit it — a published key gets auto-revoked and breaks the pipeline for everyone at once.

Apply for your own SAIA key at academiccloud.de (federated login → RPTU, add Sebastian Vollmer / dsa.dfki.de). Approval can take a week, so do it before you need it. Keys expire every 6 months.

3. Run it

Two venvs, and they are separate on purpose: module 2 (graphiti-core, neo4j) and the UI have conflicting heavy pins, so the UI shells out to module 2's interpreter rather than importing across them.

# the UI itself (streamlit + Pillow)
python3 -m venv ui/.venv
ui/.venv/bin/pip install -r ui/requirements.txt

# module 2 — the UI calls this interpreter to query the graph
python3 -m venv modules/kg-agent-memory/.venv
modules/kg-agent-memory/.venv/bin/pip install -r modules/kg-agent-memory/requirements.txt

Then either entry point:

bash scripts/run_content_map.sh --group-id gmb_finance_full --windows 2   # CLI
ui/.venv/bin/streamlit run ui/app.py                                      # UI

No venv is needed for module 3 locally — FLUX runs on unicorn.

The tunnels open themselves. Both entry points call scripts/tunnels.sh before doing anything, so the normal path is that you never think about them. (In the UI this happens when you press Generate board, not at startup.)

Where things run, and what needs a tunnel

The FLUX server is shared — one instance serves the whole team. The tunnel is per-laptop: it is your key to that shared door, so each person opens their own.

Health check

bash scripts/tunnels.sh status     # the only command worth remembering
bash scripts/tunnels.sh start      # idempotent, safe any time
bash scripts/tunnels.sh stop

🔎 Convograph tunnels via 'unicorn'
   ✅ FLUX      localhost:8500   (NVIDIA H100 PCIe)
   ✅ embedder  localhost:11435  (bge-m3)
   ℹ️  SAIA (all LLM work) and AuraDB need no tunnel.

By hand, if you prefer — both also open in a browser tab:

curl -s http://localhost:8500/health      # {"status":"warm","device":"NVIDIA H100 PCIe",...}
curl -s http://localhost:11435/api/tags   # JSON listing bge-m3

Do not use ss -ltn to check a tunnel. A dead tunnel keeps its local port bound, so ss reports it listening while every request hangs. tunnels.sh probes the services instead, and tells the two cases apart: not forwarded versus PORT BOUND BUT DEAD (which it repairs automatically).

Troubleshooting

Running on unicorn itself? Set CONVOGRAPH_NO_TUNNEL=1 — the services are already local.

Research question

How well can domain adaptation from agent memory be done from multi-party human conversation? Concretely: given what person X said over N weeks of conversation, filter the relevant subgraph and use it to adapt/ground an agent's memory — and quantify how well that adaptation works.

Pipeline

audio (4 speakers)
  -> [1] ASR + diarization  ---------------->  speaker-tagged transcript
  -> [2] temporal KG + agent memory system --> temporal KG, quantitative ablation results
  -> [3] graphic generation ----------------->  caption -> image -> editable canvas (future)

Modules are independent and communicate through the data contracts in schemas/. See docs/architecture.md for the full picture.

Modules

Adding a new module

See scripts/import_module.sh — handles both "here's a zip" and "here's a git repo" cases. Each module should stay self-contained (own README, own dependency file, own run scripts) and document how its inputs/outputs map to schemas/.

Status

Early stage, modules land independently before full end-to-end assembly. The first inter-module hop (1 -> 2) is wired and verified — run python3 pipeline/check_handoff.py (no GPU/LLM needed). A unifying UI is planned once the pipeline stabilizes (see pipeline/).

License

MIT — see LICENSE.