Jeffy

Pretrained text classifiers you can run and retrain on CPU.

Try it

Install uv, then:

uvx --python 3.12 \
  --from "jeffy-classify @ git+https://github.com/nicobrenner/jeffy.git@v0.1.0-alpha.7" \
  jeffy-serve

Open http://localhost:8400, pick a classifier, and paste one of these:

Pretrained capabilities

13 classifiers ship with the package. Weights are logistic regression coefficients (derived model parameters, not copies of training data). Source datasets and licenses are documented in ATTRIBUTION.md.

Test accuracy on held-out splits. Details in data/eval_results/benchmark.json.

Weaknesses: SNLI (65.6%) and tweet_eval_sentiment (66.2%) are below what task-specific models achieve. Emotion (75.5%) has limited class coverage. Probabilities are uncalibrated.

Train a custom classifier

uvx --python 3.12 \
  --from "jeffy-classify @ git+https://github.com/nicobrenner/jeffy.git@v0.1.0-alpha.7" \
  jeffy-train --example --save-dir my_models

Loaded 24 examples from reviews.csv
Training 'reviews': 24 examples, 2 classes
  Split: 19 train, 5 test
  Test accuracy: 100.0%
Saved to my_models/reviews/

--example uses a bundled 24-row product review CSV. To bring your own:

uvx --python 3.12 \
  --from "jeffy-classify @ git+https://github.com/nicobrenner/jeffy.git@v0.1.0-alpha.7" \
  jeffy-train --input your_data.csv --text-col text --label-col label \
  --task-id your_task --save-dir my_models

Supports .csv, .tsv, and .jsonl.

Serve a custom model

JEFFY_PACK_DIR=my_models uvx --python 3.12 \
  --from "jeffy-classify @ git+https://github.com/nicobrenner/jeffy.git@v0.1.0-alpha.7" \
  jeffy-serve

curl -s -X POST http://localhost:8400/v1/predict \
  -H "Content-Type: application/json" \
  -d '{"text": "The battery life is amazing", "task": "reviews"}'
# → {"label": "positive", "confidence": 0.87, ...}

Install from source

git clone https://github.com/nicobrenner/jeffy.git
cd jeffy

# With uv (recommended)
uv venv && uv pip install -e .

# Or with pip
python -m venv .venv && source .venv/bin/activate
pip install -e .

The first prediction downloads the shared encoder (bge-large-en-v1.5, ~1.2 GB, cached afterward).

API

Python

from jeffy.engine import Engine

engine = Engine()
engine.load()

# List available classifiers
for name, cap in engine.capabilities.items():
    print(f"{name}: {cap.description} ({cap.n_classes} classes)")

# Classify text
result = engine.predict("banking77", "I was charged twice for the same transaction")
print(result["label"])          # "transaction_charged_twice"
print(result["confidence"])     # 0.999
print(result["probabilities"])  # {"transaction_charged_twice": 0.999, ...}

SDK walkthrough

HTTP

# Banking intent
curl -s -X POST http://localhost:8400/v1/predict \
  -H "Content-Type: application/json" \
  -d '{"text": "I was charged twice for the same transaction", "task": "banking77"}'
# → {"label": "transaction_charged_twice", "confidence": 0.999, ...}

# Spam detection
curl -s -X POST http://localhost:8400/v1/predict \
  -H "Content-Type: application/json" \
  -d '{"text": "WINNER! You have been selected for a free cruise. Reply YES to claim.", "task": "sms_spam"}'
# → {"label": "spam", "confidence": 0.91, ...}

# News topic
curl -s -X POST http://localhost:8400/v1/predict \
  -H "Content-Type: application/json" \
  -d '{"text": "The Federal Reserve raised interest rates by 25 basis points on Wednesday", "task": "ag_news"}'
# → {"label": "Business", "confidence": 0.86, ...}

List capabilities

curl -s http://localhost:8400/v1/capabilities | python3 -c "
import json, sys
for c in json.load(sys.stdin)['capabilities']:
    print(f\"{c['task_id']:25s} {c['n_classes']:3d} classes  {c['test_accuracy']:.1%}  {c['name']}\")"

Per-capability metadata

Each shipped classifier has a manifest.json with label names, source dataset, HuggingFace path, stated license, encoder identity, training/test counts, and integrity hashes.

curl -s http://localhost:8400/v1/capabilities/banking77 | python3 -m json.tool

Train from Python

from jeffy.train import train_classifier

clf = train_classifier(
    texts=["great product!", "terrible service", "fast shipping", "broken on arrival"],
    labels=["positive", "negative", "positive", "negative"],
    task_id="my_reviews",
)

result = clf.predict("the quality exceeded my expectations")
print(result["label"])  # "positive"

clf.save("my_models")

Tuning options

Start with the defaults. With <50 examples per class, expect noisy estimates.

Reproduce the evaluation

# Install with build dependencies
uv pip install -e ".[build]"
# or: pip install -e ".[build]"

# Retrain all 13 heads from source datasets (~40 min, downloads ~5 GB)
jeffy-build --out data/model_pack

# Evaluate on held-out test sets with tuned baselines
jeffy-evaluate --baselines --latency --device cpu

Evaluation notes

• SST-2: Evaluated on validation split (official test labels are not public).
• SMS Spam: Random split (test_size=0.2, seed=42); no standard benchmark split.
• SNLI: Input encoded as premise [SEP] hypothesis. Label -1 filtered.
• CLINC-OOS: 151 classes including out-of-scope. In-scope accuracy 96.5%, OOS detection 51.7%.
• MASSIVE: English only (config en).

Deployment

Runtime memory: ~2 GB (encoder loaded once, shared across all heads).

Latency (CPU, single example, Linux aarch64):

Model security

Bundled pretrained artifacts use numpy .npz format (portable, no pickle). Custom-trained models also save a joblib pickle backup. Only load custom pickle artifacts from trusted sources. Each artifact's manifest.json includes integrity hashes verified on load.

Provenance and licensing

Jeffy code is MIT-licensed. Head artifacts are derived from public datasets; redistribution permissions have not been independently verified for all sources. See ATTRIBUTION.md for per-dataset license status.

The encoder (bge-large-en-v1.5) is MIT-licensed.

Tested

Verified with clean-environment wheel and sdist install on Linux aarch64, Python 3.12, scikit-learn 1.9+, sentence-transformers 6.1+, numpy 2.5+. Pretrained artifacts use numpy .npz format, avoiding sklearn version coupling.

What's not included

• No zero-shot / general classification. Each task needs a trained head. Unknown tasks return an error.
• No LLM fallback. This release is pure embedding + classifier.
• No automatic task routing. You must specify which classifier to use.

Roadmap

Suggestions for datasets, capabilities, or workflows are welcome as issues.