Dataset observability and coreset selection for robotics imitation learning
19
stars
149
commits
Python
primary language
Sep 10, 2026
updated
README
Calibra
Train robot policies with up to 75% less data.
Calibra helps robotics teams build smaller, higher-quality training sets — catching bad demonstrations before they waste GPU time, then selecting the episodes that actually matter.
Results
| Dataset | Quality Score | Best Retention | Result |
|---|---|---|---|
PushT (lerobot/pusht) | 76.7 | 25% | 99.5% of full-data performance with 75% less training data |
DROID-100 (lerobot/droid_100) | 77.0 | 75% | Outperformed full-data baseline (+3%) |
ALOHA sim (lerobot/aloha_sim_insertion_human) | 87.3 | Higher | Smaller gains — already a clean dataset |
xArm lift (lerobot/xarm_lift_medium) | 82.7 | — | Little benefit — already a high-quality simulation dataset |
Across four public robotics datasets, Calibra consistently preserved more rare behaviors than random selection. The magnitude of training-data reduction depended on the dataset's quality and redundancy.
Across three datasets and three policy families (BC-MLP, ACT, Diffusion Policy) at 30% retention, Calibra improves over random by +24.5% on average.
→ Full benchmark results, ablation tables, and limitations
How it works
The reason Calibra can remove 75% of demonstrations without hurting performance is that most robotics datasets contain two distinct problems: bad episodes (jerk spikes, dropped frames, sync errors) and redundant episodes (near-duplicate demonstrations of the same behavior). Calibra removes both.
The pipeline:
| Step | Question | Command |
|---|---|---|
| 1. Integrity | Can I trust this dataset? | calibra integrity |
| 2. Quality | Which episodes are clean? | calibra audit |
| 3. Coverage | Which episodes are distinct? | calibra review |
| 4. Select | Keep only what matters. | calibra prune |
$ calibra integrity /data/my_demos.h5
─── Dataset Integrity ────────────────────────────────────
my_demos · 120 episodes
Critical (1)
❌ camera_freeze_events: 1 of 120 episodes (0.8%) contain a run of ≥5
consecutive near-identical camera frames (episode ep_17).
Warnings (1)
⚠️ blurry_episode_fraction: camera frames markedly blurrier than the
rest of the dataset in 1 episode.
Passed (8)
✅ timestamp_jitter_cv ✅ timestamp_dropout_rate ✅ short_episode_fraction
✅ action_dropout_rate ✅ duplicate_frame_rate ✅ ldlj
✅ jerk_spike_rate ✅ velocity_discontinuity_rate
Integrity Score: 85/100 · Status: Warning
Or run all four steps as one report with calibra analyze — integrity, Calibra Score, estimated redundancy, and a training-set recommendation from the same coreset selector calibra prune uses:
$ calibra analyze lerobot/pusht
────────────────────────────────────────────────────────────
CALIBRA ANALYSIS
────────────────────────────────────────────────────────────
Dataset
Name : lerobot/pusht
Episodes : 206
...
Quality (Calibra Score) 76.7 / 100 — Good
Coverage / diversity 68.2 / 100
Redundancy (estimated) 41.0% of state-space occupies duplicate regions
──────────────────────────────────────────────────────────
RECOMMENDATION
Training set : 52 / 206 episodes
Expected retention : 25%
...
This is a heuristic starting point, not a validated retention curve.
Run the design-partner protocol (`calibra experiment` + `calibra
case-study`) before committing a production training run to this number.
Quick start
pip install calibra-robotics
calibra integrity /data/my_demos.h5
calibra audit lerobot/pusht
calibra prune lerobot/pusht --keep 0.25 --report results/pusht/latest.json
# or the whole pipeline in one command:
calibra analyze lerobot/pusht
Try it online
No installation required.
🔗 Calibra — Dataset Integrity (Hugging Face Space)
- Check any LeRobot dataset's integrity — timestamps, sync, completeness, duplicate/frozen/blurry frames, jittery motion
- See its Quality & Coverage score and percentile
- Compare against community benchmarks
- Download a full audit report
Benchmark details
On real PushT data: at 10% retention, Calibra achieves lower prediction error than training on the full dataset, while random selection degrades sharply.
Ablation across 5 seeds on ALOHA mobile (keep 30%): Calibra full pipeline and diversity-only both outperform all published baselines.
Mean improvement over random selection (5 seeds, 30% retention, 3 datasets):
| Method | BC-MLP | ACT | Diffusion Policy |
|---|---|---|---|
| Diversity-only | +29.5% | +26.5% | +11.9% |
| Calibra full | +24.5% | +23.7% | +13.8% |
| K-Center | +24.0% | +23.1% | +10.1% |
| Facility Location | +21.5% | +18.4% | +8.7% |
| Random | 0.0% | 0.0% | 0.0% |
Method rankings are stable across all three policy families (Spearman ρ ≥ 0.86).
→ Full benchmarks and ablations
Measure real training savings
Calibra can record measured training results from real experiments and connect them to benchmark reports.
calibra experiment record --experiment-id my-run --condition calibra --retention 25 \
--gpu-hours 6.2 --eval-success-rate 0.88
calibra experiment list --experiment-id my-run
calibra experiment report --experiment-id my-run
Run a retention sweep:
calibra benchmark --sweep
Connect measured results to the benchmark:
calibra benchmark --sweep --experiment-id my-run
Reports distinguish simulated, partially measured, and validated case-study results so estimated compute savings are not confused with measured results.
Once a design partner's retention curve is fully recorded, turn it into a partner-facing report:
calibra case-study --experiment-id my-run --partner "Partner A" --gpu-cost-per-hour 2.50 --out case_study.md
calibra case-study reads only real measured calibra experiment record data — never calibra benchmark's simulated numbers — and marks the report DRAFT rather than VALIDATED if any protocol condition is still unrecorded.
Why diversity-aware selection beats random
Random selection picks a clustered subset. Calibra's coverage-based selector spreads selections across the behavioral space — ensuring the policy sees every behavioral mode, even rare ones.
Dashboard
Inspect dataset health, identify problematic demonstrations with root causes, and generate a training-ready coreset — all from one interface. Generated with calibra audit lerobot/columbia_cairlab_pusht_real --html-out report.html.
In practice
LeRobot integration
# 1. Record demos
lerobot-record --robot-type so100 --repo-id $HF_USER/my_dataset
# 2. Curate and write the report
calibra prune /path/to/my_dataset --keep 0.3 --report results/my_dataset/latest.json
# 3. Train on the coreset
lerobot-train policy=act dataset_repo_id=./my_dataset_coreset
from calibra.integrations.lerobot import load_dataset
ds = load_dataset("lerobot/pusht", report_path="results/pusht/latest.json")
# ds is a datasets.Dataset with only Calibra-approved episodes
Isaac Lab → GR00T (NVIDIA)
from calibra.integrations.isaac_lab import export_gr00t_manifest, filter_hdf5
export_gr00t_manifest("results/franka/latest.json", demos_path="demos.hdf5")
filter_hdf5("demos.hdf5", "results/franka/latest.json", "demos_coreset.hdf5")
calibra prune demos.hdf5 --keep 0.3 --policy gr00t --report results/franka/latest.json
python -m gr00t.train --manifest gr00t_manifest.json --demo-file demos_coreset.hdf5
Python API
from calibra.ingestion.registry import load
from calibra.pipeline import Pipeline
from calibra.pruning import CoresetSelector
batch = load("lerobot/pusht")
report = Pipeline().run(batch, policy_family="diffusion")
selector = CoresetSelector(keep_fraction=0.3)
result = selector.select(batch, report)
# result.keep_episode_ids → filter your dataset
Commands
| Command | Description |
|---|---|
calibra analyze | One-command report: integrity, Calibra Score, estimated redundancy, and a training-set recommendation |
calibra integrity | "Can I trust this dataset?" — timestamps, sync, episode completeness, duplicate/frozen/blurry camera frames, jittery/jerky motion (--decode-images for LeRobot v1) |
calibra audit | Full diagnostic report with bootstrap CIs and per-episode outlier detection |
calibra review | Ranked episode review queue — separates anomaly, quality-risk, and coverage-value signals |
calibra prune | Two-stage coreset: quality filter + greedy max-coverage selection |
calibra certify | Structured CERTIFIED / PROVISIONAL / NOT CERTIFIED; --json for CI |
calibra predict | Estimate training outcome before spending GPU time |
calibra watch | Real-time quality feedback during teleoperation |
calibra score | Composite 0–100 score across Quality, Synchrony, Coverage, Task Structure |
calibra compare | Evidence-backed cross-dataset comparison with falsifiable claims |
calibra corrupt | Inject synthetic corruptions to validate metric sensitivity |
calibra card | Generate a HuggingFace dataset quality card |
calibra sim2real | Quantify sim-to-real distribution gap and transfer risk |
calibra transfer | Cross-embodiment compatibility scoring |
calibra cure | Automatic data remediation (smoothing, resampling, trimming) |
calibra audit-all | Bulk-audit an entire HF org; writes CalibraReport JSONs |
calibra site | Generate a static leaderboard website from audit results |
calibra serve | Local REST API server and web dashboard |
calibra benchmark | Compare full, random, and Calibra-selected datasets across training-data retention levels (--sweep, --experiment-id) |
calibra experiment | Record and report measured training results such as GPU-hours and evaluation success |
calibra case-study | Render a fully (or partially) measured experiment into a partner-facing case-study report |
Roadmap
v0.8.0 (current) — Measured training results: calibra experiment record/list/report logs a design partner's real training-run outcomes; calibra benchmark --sweep and --experiment-id fold measured numbers into the benchmark report wherever available, tagging every value (measured) or (simulated) and stamping the report SIMULATED / PARTIAL MEASUREMENT / CASE STUDY / VALIDATED. Full details in CHANGELOG.md.
Since v0.8.0 — One-command report: calibra analyze composes integrity, Calibra Score, and the coreset recommendation into a single report, and calibra case-study turns a completed experiment log into a partner-facing markdown report.
Next — Vision Integrity for video-backed LeRobot (v2/v3): decode sampled frames from LeRobot's mp4-encoded v2/v3 datasets so duplicate-frame/camera-freeze/blur detection work there too.
Install
PyPI package name:
calibra-robotics(thecalibraname on PyPI is an unrelated package)
pip install calibra-robotics # core (numpy + pydantic only)
pip install 'calibra-robotics[lerobot]' # LeRobot / HuggingFace Hub (recommended)
pip install 'calibra-robotics[hdf5]' # HDF5 (Isaac Lab, Robomimic)
pip install 'calibra-robotics[rlds]' # RLDS / TF Datasets
pip install 'calibra-robotics[mcap]' # MCAP / ROS2 bags
pip install 'calibra-robotics[all]' # everything
Formats supported: LeRobot v1/v2/v3 (Parquet), HuggingFace Hub IDs, HDF5 (Isaac Lab, Robomimic), RLDS/TF Datasets, MCAP/ROS2 bags.
Camera-frame checks (duplicate_frame_rate, camera_freeze_events, blurry_episode_fraction in calibra integrity) work out of the box on HDF5/Isaac Lab/robomimic data, and on LeRobot v1 datasets via calibra integrity <path> --decode-images (opt-in — decodes HuggingFace Image-feature columns, off by default since it increases load time/memory). Not yet supported for LeRobot v2/v3 (video-encoded).
Paper
Coming soon. The central empirical finding — that the optimal coreset selection strategy depends on the data-retention budget — will be described in full detail.
Citation
Available after paper release.
Contributing
Calibra is not open to external pull requests or contributions at this time.
Development
git clone https://github.com/omertt27/Calibra
pip install -e '.[all,dev]'
pytest # 770 tests
ruff check . # zero errors expected
License
Business Source License 1.1 — free for research and internal use, converts to Apache 2.0 on 2030-06-30. Commercial hosting requires a separate license. See LICENSE and LICENSING.md. Contact: omertahtaci05@gmail.com
Contributors
149 commits
See what people are saying
Similar repos
Languages
Python
77.0%
Jupyter Notebook
19.8%
TeX
1.5%
Dataset observability and coreset selection for robotics imitation learning
19
stars
149
commits
Python
primary language
Sep 10, 2026
updated
README
Calibra
Train robot policies with up to 75% less data.
Calibra helps robotics teams build smaller, higher-quality training sets — catching bad demonstrations before they waste GPU time, then selecting the episodes that actually matter.
Results
| Dataset | Quality Score | Best Retention | Result |
|---|---|---|---|
PushT (lerobot/pusht) | 76.7 | 25% | 99.5% of full-data performance with 75% less training data |
DROID-100 (lerobot/droid_100) | 77.0 | 75% | Outperformed full-data baseline (+3%) |
ALOHA sim (lerobot/aloha_sim_insertion_human) | 87.3 | Higher | Smaller gains — already a clean dataset |
xArm lift (lerobot/xarm_lift_medium) | 82.7 | — | Little benefit — already a high-quality simulation dataset |
Across four public robotics datasets, Calibra consistently preserved more rare behaviors than random selection. The magnitude of training-data reduction depended on the dataset's quality and redundancy.
Across three datasets and three policy families (BC-MLP, ACT, Diffusion Policy) at 30% retention, Calibra improves over random by +24.5% on average.
→ Full benchmark results, ablation tables, and limitations
How it works
The reason Calibra can remove 75% of demonstrations without hurting performance is that most robotics datasets contain two distinct problems: bad episodes (jerk spikes, dropped frames, sync errors) and redundant episodes (near-duplicate demonstrations of the same behavior). Calibra removes both.
The pipeline:
| Step | Question | Command |
|---|---|---|
| 1. Integrity | Can I trust this dataset? | calibra integrity |
| 2. Quality | Which episodes are clean? | calibra audit |
| 3. Coverage | Which episodes are distinct? | calibra review |
| 4. Select | Keep only what matters. | calibra prune |
$ calibra integrity /data/my_demos.h5
─── Dataset Integrity ────────────────────────────────────
my_demos · 120 episodes
Critical (1)
❌ camera_freeze_events: 1 of 120 episodes (0.8%) contain a run of ≥5
consecutive near-identical camera frames (episode ep_17).
Warnings (1)
⚠️ blurry_episode_fraction: camera frames markedly blurrier than the
rest of the dataset in 1 episode.
Passed (8)
✅ timestamp_jitter_cv ✅ timestamp_dropout_rate ✅ short_episode_fraction
✅ action_dropout_rate ✅ duplicate_frame_rate ✅ ldlj
✅ jerk_spike_rate ✅ velocity_discontinuity_rate
Integrity Score: 85/100 · Status: Warning
Or run all four steps as one report with calibra analyze — integrity, Calibra Score, estimated redundancy, and a training-set recommendation from the same coreset selector calibra prune uses:
$ calibra analyze lerobot/pusht
────────────────────────────────────────────────────────────
CALIBRA ANALYSIS
────────────────────────────────────────────────────────────
Dataset
Name : lerobot/pusht
Episodes : 206
...
Quality (Calibra Score) 76.7 / 100 — Good
Coverage / diversity 68.2 / 100
Redundancy (estimated) 41.0% of state-space occupies duplicate regions
──────────────────────────────────────────────────────────
RECOMMENDATION
Training set : 52 / 206 episodes
Expected retention : 25%
...
This is a heuristic starting point, not a validated retention curve.
Run the design-partner protocol (`calibra experiment` + `calibra
case-study`) before committing a production training run to this number.
Quick start
pip install calibra-robotics
calibra integrity /data/my_demos.h5
calibra audit lerobot/pusht
calibra prune lerobot/pusht --keep 0.25 --report results/pusht/latest.json
# or the whole pipeline in one command:
calibra analyze lerobot/pusht
Try it online
No installation required.
🔗 Calibra — Dataset Integrity (Hugging Face Space)
- Check any LeRobot dataset's integrity — timestamps, sync, completeness, duplicate/frozen/blurry frames, jittery motion
- See its Quality & Coverage score and percentile
- Compare against community benchmarks
- Download a full audit report
Benchmark details
On real PushT data: at 10% retention, Calibra achieves lower prediction error than training on the full dataset, while random selection degrades sharply.
Ablation across 5 seeds on ALOHA mobile (keep 30%): Calibra full pipeline and diversity-only both outperform all published baselines.
Mean improvement over random selection (5 seeds, 30% retention, 3 datasets):
| Method | BC-MLP | ACT | Diffusion Policy |
|---|---|---|---|
| Diversity-only | +29.5% | +26.5% | +11.9% |
| Calibra full | +24.5% | +23.7% | +13.8% |
| K-Center | +24.0% | +23.1% | +10.1% |
| Facility Location | +21.5% | +18.4% | +8.7% |
| Random | 0.0% | 0.0% | 0.0% |
Method rankings are stable across all three policy families (Spearman ρ ≥ 0.86).
→ Full benchmarks and ablations
Measure real training savings
Calibra can record measured training results from real experiments and connect them to benchmark reports.
calibra experiment record --experiment-id my-run --condition calibra --retention 25 \
--gpu-hours 6.2 --eval-success-rate 0.88
calibra experiment list --experiment-id my-run
calibra experiment report --experiment-id my-run
Run a retention sweep:
calibra benchmark --sweep
Connect measured results to the benchmark:
calibra benchmark --sweep --experiment-id my-run
Reports distinguish simulated, partially measured, and validated case-study results so estimated compute savings are not confused with measured results.
Once a design partner's retention curve is fully recorded, turn it into a partner-facing report:
calibra case-study --experiment-id my-run --partner "Partner A" --gpu-cost-per-hour 2.50 --out case_study.md
calibra case-study reads only real measured calibra experiment record data — never calibra benchmark's simulated numbers — and marks the report DRAFT rather than VALIDATED if any protocol condition is still unrecorded.
Why diversity-aware selection beats random
Random selection picks a clustered subset. Calibra's coverage-based selector spreads selections across the behavioral space — ensuring the policy sees every behavioral mode, even rare ones.
Dashboard
Inspect dataset health, identify problematic demonstrations with root causes, and generate a training-ready coreset — all from one interface. Generated with calibra audit lerobot/columbia_cairlab_pusht_real --html-out report.html.
In practice
LeRobot integration
# 1. Record demos
lerobot-record --robot-type so100 --repo-id $HF_USER/my_dataset
# 2. Curate and write the report
calibra prune /path/to/my_dataset --keep 0.3 --report results/my_dataset/latest.json
# 3. Train on the coreset
lerobot-train policy=act dataset_repo_id=./my_dataset_coreset
from calibra.integrations.lerobot import load_dataset
ds = load_dataset("lerobot/pusht", report_path="results/pusht/latest.json")
# ds is a datasets.Dataset with only Calibra-approved episodes
Isaac Lab → GR00T (NVIDIA)
from calibra.integrations.isaac_lab import export_gr00t_manifest, filter_hdf5
export_gr00t_manifest("results/franka/latest.json", demos_path="demos.hdf5")
filter_hdf5("demos.hdf5", "results/franka/latest.json", "demos_coreset.hdf5")
calibra prune demos.hdf5 --keep 0.3 --policy gr00t --report results/franka/latest.json
python -m gr00t.train --manifest gr00t_manifest.json --demo-file demos_coreset.hdf5
Python API
from calibra.ingestion.registry import load
from calibra.pipeline import Pipeline
from calibra.pruning import CoresetSelector
batch = load("lerobot/pusht")
report = Pipeline().run(batch, policy_family="diffusion")
selector = CoresetSelector(keep_fraction=0.3)
result = selector.select(batch, report)
# result.keep_episode_ids → filter your dataset
Commands
| Command | Description |
|---|---|
calibra analyze | One-command report: integrity, Calibra Score, estimated redundancy, and a training-set recommendation |
calibra integrity | "Can I trust this dataset?" — timestamps, sync, episode completeness, duplicate/frozen/blurry camera frames, jittery/jerky motion (--decode-images for LeRobot v1) |
calibra audit | Full diagnostic report with bootstrap CIs and per-episode outlier detection |
calibra review | Ranked episode review queue — separates anomaly, quality-risk, and coverage-value signals |
calibra prune | Two-stage coreset: quality filter + greedy max-coverage selection |
calibra certify | Structured CERTIFIED / PROVISIONAL / NOT CERTIFIED; --json for CI |
calibra predict | Estimate training outcome before spending GPU time |
calibra watch | Real-time quality feedback during teleoperation |
calibra score | Composite 0–100 score across Quality, Synchrony, Coverage, Task Structure |
calibra compare | Evidence-backed cross-dataset comparison with falsifiable claims |
calibra corrupt | Inject synthetic corruptions to validate metric sensitivity |
calibra card | Generate a HuggingFace dataset quality card |
calibra sim2real | Quantify sim-to-real distribution gap and transfer risk |
calibra transfer | Cross-embodiment compatibility scoring |
calibra cure | Automatic data remediation (smoothing, resampling, trimming) |
calibra audit-all | Bulk-audit an entire HF org; writes CalibraReport JSONs |
calibra site | Generate a static leaderboard website from audit results |
calibra serve | Local REST API server and web dashboard |
calibra benchmark | Compare full, random, and Calibra-selected datasets across training-data retention levels (--sweep, --experiment-id) |
calibra experiment | Record and report measured training results such as GPU-hours and evaluation success |
calibra case-study | Render a fully (or partially) measured experiment into a partner-facing case-study report |
Roadmap
v0.8.0 (current) — Measured training results: calibra experiment record/list/report logs a design partner's real training-run outcomes; calibra benchmark --sweep and --experiment-id fold measured numbers into the benchmark report wherever available, tagging every value (measured) or (simulated) and stamping the report SIMULATED / PARTIAL MEASUREMENT / CASE STUDY / VALIDATED. Full details in CHANGELOG.md.
Since v0.8.0 — One-command report: calibra analyze composes integrity, Calibra Score, and the coreset recommendation into a single report, and calibra case-study turns a completed experiment log into a partner-facing markdown report.
Next — Vision Integrity for video-backed LeRobot (v2/v3): decode sampled frames from LeRobot's mp4-encoded v2/v3 datasets so duplicate-frame/camera-freeze/blur detection work there too.
Install
PyPI package name:
calibra-robotics(thecalibraname on PyPI is an unrelated package)
pip install calibra-robotics # core (numpy + pydantic only)
pip install 'calibra-robotics[lerobot]' # LeRobot / HuggingFace Hub (recommended)
pip install 'calibra-robotics[hdf5]' # HDF5 (Isaac Lab, Robomimic)
pip install 'calibra-robotics[rlds]' # RLDS / TF Datasets
pip install 'calibra-robotics[mcap]' # MCAP / ROS2 bags
pip install 'calibra-robotics[all]' # everything
Formats supported: LeRobot v1/v2/v3 (Parquet), HuggingFace Hub IDs, HDF5 (Isaac Lab, Robomimic), RLDS/TF Datasets, MCAP/ROS2 bags.
Camera-frame checks (duplicate_frame_rate, camera_freeze_events, blurry_episode_fraction in calibra integrity) work out of the box on HDF5/Isaac Lab/robomimic data, and on LeRobot v1 datasets via calibra integrity <path> --decode-images (opt-in — decodes HuggingFace Image-feature columns, off by default since it increases load time/memory). Not yet supported for LeRobot v2/v3 (video-encoded).
Paper
Coming soon. The central empirical finding — that the optimal coreset selection strategy depends on the data-retention budget — will be described in full detail.
Citation
Available after paper release.
Contributing
Calibra is not open to external pull requests or contributions at this time.
Development
git clone https://github.com/omertt27/Calibra
pip install -e '.[all,dev]'
pytest # 770 tests
ruff check . # zero errors expected
License
Business Source License 1.1 — free for research and internal use, converts to Apache 2.0 on 2030-06-30. Commercial hosting requires a separate license. See LICENSE and LICENSING.md. Contact: omertahtaci05@gmail.com
See what people are saying
Similar repos
Contributors
149 commits
Languages
Python
77.0%
Jupyter Notebook
19.8%
TeX
1.5%