WaterSheep

Website · Demo · Code

WaterSheep answers yes/no, single-choice, rating and multi-label questions about any text, with a probability for every option. Version 0.1.0 (watersheep-20260928-125452).

Usage

pip install transformers torch

from transformers import pipeline

ws = pipeline(model="samratduttaofficial/WaterSheep", trust_remote_code=True)
ws("I was charged twice.", question="Which team should handle this?", options=["billing", "shipping", "support"])

Every answer includes a probability for each option.

Download

hf download samratduttaofficial/WaterSheep --local-dir WaterSheep

Or with Git (requires Git LFS):

git clone https://huggingface.co/samratduttaofficial/WaterSheep

Then load it from the folder, offline:

ws = pipeline(model="WaterSheep", trust_remote_code=True)

API

Deploy it as an Inference Endpoint, then:

curl https://YOUR-ENDPOINT -H "Authorization: Bearer $HF_TOKEN" -H "Content-Type: application/json" -d '{"inputs": "I was charged twice.", "parameters": {"question": "Which team should handle this?", "options": ["billing", "shipping", "support"]}}'

JavaScript

No install; runs in the browser:

<script type="module">
  import { decide } from "https://samratduttaofficial.github.io/WaterSheep/watersheep.js";
  console.log(await decide("I was charged twice.", "Which team should handle this?", ["billing", "shipping", "support"]));
</script>

With a downloaded copy on your web server, call load({ base: "WaterSheep/" }) first.

Other languages: run onnx/model_quantized.onnx with ONNX Runtime; watersheep.js shows the input format.

Evaluation

ECE is the expected calibration error (lower is better).

Accuracy against confidence for each question type, before (raw) and after calibration.

Benchmarks

Training

Training loss and learning rate (left); validation accuracy by question type (right).

Share of synthetic examples kept after verification, by question type (left) and by family (right).

Limitations

• English only.
• Long inputs are truncated.
• Rating-scale answers are less accurate than the other types.
• Probabilities are calibrated on data like the training data; validate them on your own.
• Not for high-stakes decisions (medical, legal, financial, hiring) on its own.

License

Apache 2.0 (LICENSE). Trained on openly licensed data; credits in NOTICE.

Citation

@misc{watersheep,
  author = {Samrat Dutta},
  title  = {WaterSheep: calibrated decisions for any text},
  year   = {2026},
  url    = {https://huggingface.co/samratduttaofficial/WaterSheep}
}

---