AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents

This repository hosts the task streams used in the paper AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents.

Stream Types

AgentCL organizes tasks into streams according to their cross-task relation:

• Conventional: tasks come from the same environment or domain, but whether reusable knowledge exists between them is unknown.
• Dependent: the stream simulates capability-centered progression, where later tasks can reuse knowledge from earlier ones. Each complex task can reuse the solution to an earlier subtask.
• Independent: held-out tasks with little transferable knowledge by design, evaluated after a conventional or dependent stream. For example, the HumanEval-Pro streams serve as independent tasks after the BigCodeBench-Lite-Pro streams.

The order of tasks within a file is the stream order.

Notes:

• agentboard_scienceworld/block.jsonl contains the same 90 tasks as conventional.jsonl, grouped by near-identical goal families instead. It is used for the block-stream analysis in the paper.
• In the CodeEval-Pro dependent files, category: "raw" marks a subtask and category: "new" marks a complex task.
• In BrowseComp-Plus, *_subqueries.* contain the 308 subtasks alone, which were synthesized from the 100 parent tasks. query_type and parent_query_id link each subtask to its parent.
• The queries*.tsv files have an id/task header. A few original task texts contain tab characters, so split each line on the first tab only, or read the same queries from ground_truth*.jsonl.

IDs

• In the CodeEval-Pro dependent files, a subtask and its complex task share the same id, so (category, id) identifies a record. corresponding_new_id (on a subtask) and corresponding_raw_id (on a complex task) point to its pair.
• In agentboard_babyai/dependent.jsonl, dependent_pair_source_id and dependent_pair_target_id refer to the id of the paired subtask and complex task.

Data Sources

• agentboard_babyai, agentboard_scienceworld, and mmlu_pro are included as ready-to-use subsets. They are direct subsets from existing public datasets, with no modifications to the original data.
• Our constructed codeeval-pro and browsecomp_plus streams are provided.

The metadata in source datasets only indicates the authorship for the source datasets, and does not imply the authorship of this repository. The source cards remain the authority for dataset terms and licenses.

CodeEval-Pro

We build CodeEval-Pro files from the following sources:

• Paired-task inputs:

raw_public/humaneval_pro.json
  https://huggingface.co/datasets/CodeEval-Pro/humaneval-pro

raw_public/mbpp_pro.json
  https://huggingface.co/datasets/CodeEval-Pro/mbpp-pro

raw_public/bigcodebench_lite_pro.json
  https://huggingface.co/datasets/CodeEval-Pro/bigcodebench-lite-pro

• Raw-task test-code inputs:

raw_test_code/humaneval-raw-test-json
  https://huggingface.co/datasets/openai/openai_humaneval

raw_test_code/mbpp-raw-test-json
  https://huggingface.co/datasets/evalplus/mbppplus

raw_test_code/bigcodebench-raw-test-json
  https://huggingface.co/datasets/bigcode/bigcodebench

Citation

@article{shu2026agentcl,
  title={AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents},
  author={Shu, Yiheng and Guti{\'e}rrez, Bernal Jim{\'e}nez and Jonnalagedda, Saisri Padmaja and Yao, Yuguang and Sun, Huan and Su, Yu},
  journal={arXiv preprint arXiv:2606.02461},
  year={2026}
}