NeuralKV — Local Knowledge-Injected Transformer

Vision

A Rust-based transformer inference engine with NeuralDB-style knowledge injection, designed for local deployment on consumer GPUs (RTX 2080, 8GB VRAM). Primary use case: intelligent email management with persistent, editable domain knowledge.

Core Concept

Instead of relying on large cloud models with conversational context, use a small distilled model (e.g., QwOpus — Qwen distilled with Opus reasoning) and inject task-specific knowledge directly into the model's weights via a neural Key-Value database. The model "just knows" your domain without needing prompts or RAG.

Research Foundation

NeuralDB: Scaling Knowledge Editing in LLMs to 100,000 Facts

• Paper: https://openreview.net/forum?id=Z0CX62CSJQ
• arXiv: https://arxiv.org/abs/2507.18028
• Authors: Weizhi Fei, Hao Shi, Jing Xu, Jingchen Peng, Jiazheng Li, Jingzhao Zhang, Bo Bai, Wei Han, Zhenyuan Chen, Xueyan Niu
• Key contribution: Reframes Locate-and-Edit methods as querying a KV database, then replaces linear retrieval with a non-linear gated retrieval module. Scales to 100K edited facts (50x prior work) without degrading general model performance.
• Tested on: GPT2-XL, GPT-J (6B), Llama-3 (8B)
• Benchmarks: ZsRE, CounterFact datasets; preserves performance on 6 representative text understanding/generation benchmarks

Why This Matters

Existing approaches to injecting knowledge into LLMs:

• Fine-tuning: Expensive, catastrophic forgetting, needs retraining for each update
• RAG: Retrieval latency, context window limits, no true "understanding"
• Prompt engineering: Context window limits, knowledge lost between sessions
• NeuralDB: Knowledge becomes part of the model's weights via KV entries, editable at any time, scales to 100K facts, no degradation

Architecture (Proposed)

+-------------------+     +------------------+     +------------------+
|  Rust Inference   |     |   NeuralDB KV    |     |   Knowledge      |
|  Engine           |<--->|   Store (Gated    |<--->|   Editor         |
|  (quantized model)|     |   Retrieval)     |     |   (CLI/API)      |
+-------------------+     +------------------+     +------------------+
         |                                                  |
         v                                                  v
+-------------------+                              +------------------+
|  Task Interface   |                              |  Learning Loop   |
|  (Email/Outlook   |                              |  (User feedback  |
|   add-in, CLI)    |                              |   -> KV entries) |
+-------------------+                              +------------------+

Components

1. Rust Inference Engine

   • Quantized model execution (Q4/Q5 on 8GB VRAM)
   • Base model: QwOpus or similar distilled reasoning model
   • CUDA/cuDNN for RTX 2080

2. NeuralDB KV Store

   • Non-linear gated retrieval module (per the paper)
   • Persistent storage of knowledge entries
   • Hot-swappable — edit/add/remove facts without reloading

3. Knowledge Editor

   • CLI or API for injecting facts
   • Example entries for email use case:
     • ("aj@jaiglo.com", "legitimate IT provider for GRM, never junk")
     • ("elena@jaiglo.com", "spam newsletter, always junk")
     • ("dse_NA4@docusign.net", "phishing campaign impersonating Docusign, always junk")
     • ("INBOX/Junk Email", "correct Outlook junk folder, not Spam")

4. Task Interface

   • Email: IMAP connection, Outlook add-in, or standalone
   • Presents ranked decisions to user
   • User approve/deny feeds back into KV store

5. Learning Loop

   • User corrections become new KV entries automatically
   • No retraining needed — just insert into the neural database
   • Corrections persist across sessions

Target Hardware

• GPU: NVIDIA RTX 2080 (8GB VRAM)
• Model: 7B parameters, Q4 quantization (~4GB VRAM)
• NeuralDB overhead: ~1-2GB additional
• Inference speed: Real-time classification (~100ms per email)

Email Use Case — What We Learned

From a real email cleanup session (2026-03-20), managing dan.u@grmcompany.com:

• 310 spam emails moved to junk across 86+ senders
• 70 one-click unsubscribes executed
• Key insight: The AI correctly classified ~80% of emails, but the remaining 20% required human domain knowledge (e.g., Docusign phishing looked legitimate, same domain had both legit and spam senders)
• A NeuralDB approach would let those human corrections persist as weight-level knowledge, not just memory files

Related Models

• QwOpus — Qwen distilled with Claude Opus reasoning (community model)
• Qwen 2 7B — fits on RTX 2080 with Q4 quantization
• Llama 3 8B — tested in the NeuralDB paper

Status

Research/planning phase. See the paper for implementation details of the gated retrieval module.