Kompress API Server

FastAPI server wrapping chopratejas/kompress-v2-base for token-level importance scoring and context compression.

Drop low-scoring tokens before sending prompts to an LLM. The model is a ModernBERT-base + LoRA + span-conv head, exported as INT8 ONNX (~261 MB) for fast CPU inference.

---

✨ Features

• POST /compress — keep top-N% most important tokens, return reconstructed text
• POST /score — return per-token importance scores as JSON
• POST /reload — switch INT8 ↔ FP32 at runtime
• GET /health — model status + load time
• CORS enabled (works from local browser clients)
• Lazy-loaded, cached ONNX session (single-shot model warmup)
• ~250 ms for 400-token inputs on Apple Silicon CPU

---

📦 Installation

# 1. Create venv
python -m venv ~/venvs/kompress
source ~/venvs/kompress/bin/activate

# 2. Install this package (editable)
cd kompress-api-server
pip install -e .

# 3. Download the model (INT8 only — ~261 MB)
huggingface-cli download chopratejas/kompress-v2-base \
  --include "onnx/kompress-int8-wo.onnx" "*tokenizer*" "*config*"

If you prefer FP32 (~573 MB, slightly more accurate):

huggingface-cli download chopratejas/kompress-v2-base \
  --include "onnx/*.onnx" "*tokenizer*" "*config*"

---

🚀 Run

# Option 1: CLI entry point (after `pip install -e .`)
kompress-api-server --port 7777

# Option 2: uvicorn directly
python -m uvicorn kompress_api.server:app --host 0.0.0.0 --port 7777 --reload

# Option 3: programmatic
python -c "from kompress_api.server import app; import uvicorn; uvicorn.run(app, port=7777)"

OpenAPI docs at http://localhost:7777/docs.

---

🔌 API

GET /health

{
  "status": "ok",
  "version": "1.0.0",
  "model": "chopratejas/kompress-v2-base",
  "quant": "int8",
  "model_loaded": true,
  "load_time_s": 0.34
}

POST /compress

curl -X POST http://localhost:7777/compress \
  -H "Content-Type: application/json" \
  -d '{
    "text": "The quick brown fox jumps over the lazy dog.",
    "keep_ratio": 0.5,
    "max_length": 512
  }'

Response:

{
  "compressed": " quick brown fox over lazy dog",
  "n_total": 12,
  "n_kept": 8,
  "kept_pct": 66.7
}

POST /score

curl -X POST http://localhost:7777/score \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello world from Kompress API", "max_length": 64}'

Response:

{
  "tokens": [
    {"i": 0, "tok": "[CLS]", "id": 50281, "score": 0.605244, "special": true},
    {"i": 1, "tok": "Hello", "id": 12092, "score": 0.907625, "special": false},
    {"i": 2, "tok": "\u0120world", "id": 1533, "score": 0.970307, "special": false},
    ...
  ]
}

POST /reload?quant=fp32

Swap quantization at runtime:

curl -X POST "http://localhost:7777/reload?quant=fp32"

---

🐍 Python client

import httpx

with httpx.Client(base_url="http://localhost:7777") as c:
    r = c.post("/compress", json={
        "text": "long prompt here...",
        "keep_ratio": 0.4,
    })
    r.raise_for_status()
    print(r.json()["compressed"])

---

🧪 Tests

pip install -e ".[dev]"
pytest tests/ -v

The test suite hits a running server (set KOMPRESS_URL=http://localhost:7777).

---

🌍 Use cases

• Context compression before LLM calls — keep only the top-30% most important tokens
• Importance-aware RAG chunking — split documents by importance drops, not arbitrary windowing
• Conversation summarization preprocessing — strip boilerplate from logs before summarization
• Token-level saliency debugging — visualize which tokens a model "pays attention to"

---

⚠️ Limitations

• English-focused. kompress-v2-base is trained on English text; CJK / non-Latin input produces garbled output. For multilingual use, pair with a multilingual compression model.
• 512 tokens max by default. Longer inputs are truncated. Increase max_length up to 2048.
• CPU only by default. GPU providers can be added by editing model.load().

---

📁 Project layout

kompress-api-server/
├── pyproject.toml
├── README.md
├── src/
│   └── kompress_api/
│       ├── __init__.py
│       ├── model.py        # core ONNX + tokenizer wrapper
│       └── server.py       # FastAPI app
├── tests/
│   └── test_api.py
└── examples/
    ├── requests_examples.py
    └── basic.sh

---

📄 License

MIT.

Model: see chopratejas/kompress-v2-base for upstream terms.