Salesforce/Libra

Dataset

Libra Dataset

3 commits

1 linked in READMEs

updated May 5, 2026

See the code

README

Libra Dataset

Training and evaluation data for Libra (Libra Information Browsing and Retrieval Agent) — a self-evolving Catalog system for code localization.

Libra's core insight is optimize the environment, not the agent: instead of tuning prompts or model weights, Libra iteratively refines a hierarchical Markdown Catalog that guides a frozen locator agent's code navigation. Every localization failure drives a Catalog improvement via a Healer agent.

Dataset Structure

This dataset contains 13 configs organized into two categories:

Prompter Configs (12 repos)

Synthetic code-localization Q/A pairs generated by the Prompter agent. Each instance is created by sampling a random ~100-line code chunk from a repository and generating a realistic inquiry (bug report, feature question, behavioral clarification, etc.) that requires locating the relevant file and function.

Config	Train	Test
`prompter_astropy`	4,670	300
`prompter_django`	9,543	300
`prompter_flask`	897	99
`prompter_matplotlib`	9,042	300
`prompter_pylint`	4,601	300
`prompter_pytest`	4,694	300
`prompter_requests`	1,998	222
`prompter_seaborn`	2,401	266
`prompter_sklearn`	9,589	300
`prompter_sphinx`	4,624	300
`prompter_sympy`	34,000	300
`prompter_xarray`	4,677	300
Total	90,736	3,287

Columns:

Column	Type	Description
`instance_id`	string	Unique identifier tied to the source SWE-bench instance
`problem_statement`	string	Synthetic inquiry (bug report, feature question, etc.) generated by the Prompter
`gold_files`	string	Ground-truth file path(s) to edit
`gold_functions`	string	Ground-truth function(s) to edit
`gold_reasoning`	string	Reasoning trace explaining why this file/function is the correct target
`chunk_content`	string	The source code chunk the Prompter used to generate the question
`line_numbers`	string	Line range of the sampled chunk
`is_valid_chunk`	bool	Whether the chunk was deemed valid for question generation

SWE-bench Evaluation Config

Real bug reports from SWE-bench Lite augmented with gold file and function labels for evaluating the Locator agent.

Config	Test
`SWE-bench_Lite_Libra`	199

Columns: instance_id, repo, base_commit, problem_statement, hints_text, patch, test_patch, gold_files, gold_functions, FAIL_TO_PASS, PASS_TO_PASS, version, environment_setup_commit

Usage

from datasets import load_dataset

# Load a single repo's prompter data
ds = load_dataset("Salesforce/Libra", "prompter_sympy")
train = ds["train"]  # 29,653 instances
test = ds["test"]    # 300 instances

# Load the SWE-bench evaluation set
eval_ds = load_dataset("Salesforce/Libra", "SWE-bench_Lite_Libra", split="test")

How the Data Was Generated

The Prompter agent (Claude-based) generates each Q/A pair by:

Sampling a random code chunk from the target repository
Reading the chunk and crafting a realistic inquiry (bug report, feature question, behavioral clarification, etc.) that a developer might file
Recording the ground-truth file and function as gold labels

The Prompter has direct access to the source code (no retrieval needed), creating an information asymmetry with the Locator agent that must navigate the Catalog to find the answer. This asymmetry is the training signal — every question the Locator gets wrong reveals a Catalog deficiency for the Healer to fix.

Ethics Disclaimer

This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high-risk scenarios where errors or misuse could significantly impact people's lives, rights, or safety. For further guidance on use cases, refer to our standard AUP and AI AUP.