Foresight-Driven Retrieval for Time-Series Forecasting

Repository: https://github.com/kaichen-z/time-series

New canonical evolving-agent implementation

The collaborator's top-level evolving_loop/ package is now the base for the self-evolving experiment. The original Fresh-vs-Skill-Library baseline is preserved. The integrated version adds numbers-only program evolution with historical hindcasting, outcome-validated Retrieval and Decision skill libraries, verified contextual retrieval, a citation-constrained Decision Agent, a real Chronos ablation, and failure-attributed three-agent co-evolution with held-out acceptance. The evolve entrypoint has three general controlled levels: --evolution-mode prompt (one prompt only), genome (prompts, budgets, and topology), and source (audited Python source patches generated in isolated Git worktrees), plus the dedicated --evolution retrieval coordinate described below. See docs/EVOLVING_AGENT.md for the exact information boundaries, flow, metrics, and commands. The initial English experiment report is available at docs/EVOLUTION_METHODS_AND_RESULTS.md. A consolidated audit of all earlier baseline families and saved results is available at docs/BASELINE_METHODS_AND_RESULTS.md. For a concise map of the generic Self-Harness, dictionary curation adapter, three-agent Meta-Harness, evolution depths, and data protocol, see docs/SELF_EVOLUTION_FRAMEWORK.md. The complete installation, baseline, Coding-setting, skill-learning, and evolution manual is available at docs/USAGE_GUIDE.md.

Numerical multi-parent Combined evolution

The Numerical Agent's initial executable portfolio is 93 history-only Statistical leaves, five manifest-bound TSFMs, and five legacy-compatible Combined policies. Combined policies now support two-to-five ordered leaf parents, including TSFM–TSFM and TSFM–Statistical combinations (at least one parent must be a TSFM), with reviewed weighted_mean, median, trimmed_mean, and two-parent route operators. Each policy has an explicit successful fallback; unavailable, crashed, invalid, or not-applicable non-fallback parents use that fallback only when it is structurally valid. Duplicate parents, unknown leaves, all-Statistical combinations, Combined-to-Combined references, invalid weights, and oversized portfolios are rejected.

This is a no-weight-training, history-only flow:

history-only materialized Statistical/TSFM leaves
  -> LLM proposes strict add/repair/fork/remove Combined operations
  -> Python validates atomic Child
  -> future formal Train/Dev controller evaluates
  -> accepted canonical policy source becomes next Git generation

Task-conditioned screening derives its candidate namespace and default safety ceiling as len(module.names()) + len(portfolio.names) from the parsed Statistical module and current portfolio. The initial 93 + 5 + 5 = 103 composition is an example; accepted Combined additions increase the runtime count without changing the formal 80/20 proposal-acceptance boundary.

The current proposal adapter is not yet wired into the formal 80/20 command. It does not train or modify LLM/TSFM weights, and no performance result is claimed. Python owns schema, namespace, parent, finite-forecast, horizon, fallback, and immutable TSFM-manifest checks; proposals cannot read future labels, documents, Retrieval artifacts, runtime/checkpoint bindings, scorers, or split definitions. See numerical_agent/README.md for the portfolio contract, legacy migration behavior, and candidate-count formula (len(module.names()) + len(portfolio.names)).

Parameterized Self-Evolution framework

The repository also contains a domain-independent Parent/Child evolution core plus a numerical tool-dictionary curation adapter. The framework accepts externally supplied base methods, implementers, runtimes, tasks, metrics, and evolution parameters. It now ships an auditable 166-method / 115-source definition dataset spanning Statistical, TSFM, and Combined methods; these are method cards, not bundled executable forecasting runtimes. See numerical_agent/README.md for the component contracts, artifacts, release build command, and offline fake-provider smoke command.

An opt-in Setting 2 extension adds diagnostic-selected, source-backed forecasting guidance to the existing Coding evolution loop. See docs/setting2/README.md for the integration flow, knowledge provenance, commands, and historical results.

This repository contains an auditable, gap-guided agent loop for the Dr-CiK contextual time-series forecasting benchmark. Its research hypothesis is that a passage should be retrieved because it is expected to improve the downstream forecast—not merely because it is lexically similar to a query. A numerical forecasting backbone creates an immutable baseline; a structured retrieval controller fills explicit information gaps; and a reviser may change only the future forecast through a small, evidence-backed action language.

The numerical backbone is now Amazon Chronos-Bolt by default. The paper's forecast workspace and restricted actions are combined with this project's structured gap judging, forecast-utility retrieval, evidence grounding, and evidence-to-impact translation. The current runtime remains deterministic and exposes interfaces for learned retrievers, judges, and PostTime-style revisers. It does not claim that its label-free utility proxy is an already trained PRM.

Unified experiment command

All executable baselines and evolution levels now share one Python entrypoint. List the names with:

python -m evolving_loop --list-methods

Run a baseline by name:

python -m evolving_loop \
  --baseline chronos \
  --sample-dir external/Dr-CiK/sample \
  --task-id task_42 \
  --output-dir outputs/baselines/chronos-task42

python -m evolving_loop \
  --baseline codex-triad \
  --sample-dir external/Dr-CiK/sample \
  --task-id task_42 \
  --codex-model gpt-5.6-sol \
  --codex-reasoning-effort high \
  --output-dir outputs/baselines/codex-triad-task42

Run one evolution level by name:

python -m evolving_loop \
  --evolution genome \
  --tasks-file external/Dr-CiK/full-download/Dr-CiK_public/tasks \
  --generations 3 \
  --children 2

Run all three controlled llm_only evolution conditions with one reproducible Bash script:

scripts/run_llm_only_evolutions.sh \
  external/Dr-CiK/full-download/Dr-CiK_public/tasks all

The script gives prompt, genome, and source the same model, seed, entity split, generations, children, and timeouts while keeping every artifact in a separate subdirectory under runs/llm_only_evolution/. Use EA_DRY_RUN=1 to print the three commands without executing them.

Evolution now reserves entity-disjoint Train, Dev, and Public Holdout partitions and writes a split manifest. After freezing the artifact, run label-free Hidden Test inference separately:

python -m evolving_loop \
  --inference genome \
  --hidden-test \
  --policy-path runs/evolving/best_policy.json \
  --setting llm_only \
  --output-dir outputs/evolving-hidden

Use --inference prompt for a prompt policy or --inference source --source-patch-path runs/evolving/best_source.patch for an accepted source patch. Hidden inference writes submission-format files and never scores, learns skills, or evolves the harness.

Baseline names are skill-fresh, skill-library, chronos, timesfm, statistical, one-pass, iterative, iterative-unsafe, oracle-context, rules-triad, codex-triad, codex-direct, codex-contract, and evolving-harness. Evolution names are prompt, genome, and source. retrieval is also available as a dedicated, fixed-protocol evolution coordinate. chronos, timesfm, and statistical are true numbers-only runs: they do not retrieve or consume documents. oracle-context is restricted to public development diagnostics. The previous evolving-agent run/evolve and drcik-agent run-sample/run-hf interfaces remain supported. The two skill-* baselines use the original numbers-only JSONL interface through --tasks-file.

Two-stage Retrieval self-evolution

Implementation status as of 2026-08-28: the typed two-stage runtime, deterministic verifier, Retrieval Genome/release format, three scoped mutations, fixed 80 Train / 20 read-only Dev acceptance loop, authenticated checkpoint resume, frozen inference, and coordinate hand-off are implemented and covered by fake-LLM tests. The checked-in v000 release is an unevaluated seed (state: seed, acceptance_reason: not_evaluated_seed), not an accepted experimental result. No real Retrieval 80/20 LLM evolution has been run, no post-v000 Retrieval release has been accepted, and no Retrieval Public Regression or hidden-test score exists. The Numerical results below predate this Retrieval experiment and must not be read as Retrieval results.

The legacy and two-stage paths remain deliberately distinct:

In both paths the Numerical Agent receives only the historical numerical view. In two-stage mode, Round 1 cannot see Numerical candidates or assumptions. Round 2 receives only the host-sanitized four-field assumptions and named gaps below; it never receives candidate IDs, forecast arrays, hindcast scores, source code, future values, ground-truth evidence, or evaluator-only document roles/subtypes. The Decision Agent can select only an executed host candidate. Invalid JSON fails closed: a fatal Round 1 response preserves the pure numerical fallback, while a failed Round 2 preserves verified Round 1 evidence.

The exact safe request shapes are:

Round 1 input (exact top-level keys)
{
  "target": {
    "entity_name": str, "target_name": str, "description": str,
    "frequency": str, "forecast_window": [str, str] | []
  },
  "documents": [{"document_id": str, "content": str}],
  "retrieval_skills": [validated skill summaries]
}

Round 2 input (exact top-level keys)
{
  "target": {same safe target shape},
  "documents": [{"document_id": str, "content": str}],
  "round1": {verified RetrievalRoundResult},
  "gaps": [{
    "assumption_id": str, "gap_type": str,
    "missing_information": str, "priority": "high" | "medium" | "low"
  }],
  "assumptions": [{
    "assumption_id": str, "kind": str, "claim": str, "failure_condition": str
  }],
  "retrieval_skills": [validated skill summaries]
}

Both model stages return the same strict wire result. The first four fields are required; the last three are optional. Each EvidenceChain has exactly the fields shown here before the host checks the quote, entity, target, horizon, mechanism, magnitude, assumption identity, and budgets:

RetrievalRoundResult = {
  "evidence_chains": [EvidenceChain],
  "counterevidence": [EvidenceChain],
  "missing_information": [str],
  "sufficient": bool,
  "gaps"?: [RetrievalGap],
  "rejected"?: [str],
  "unresolved_contradictions"?: [str]
}

EvidenceChain = {
  "chain_id": str, "claim": str,
  "entity_match": bool, "target_match": bool,
  "temporal_relation": str, "mechanism": str, "direction": str,
  "magnitude_kind": str, "magnitude_value": float | null,
  "start_timestamp": str | null, "end_timestamp": str | null,
  "citations": [{"document_id": str, "exact_quote": str}],
  "missing_links": [str], "used_skill_ids": [str],
  "addressed_assumption_ids": [str], "stance": str,
  "numeric_eligible": bool
}

FinalRetrievalCard = {
  "round1": RetrievalRoundResult, "round2": RetrievalRoundResult | null,
  "chains": [EvidenceChain], "selected_document_ids": [str],
  "rejected": [str], "unresolved_contradictions": [str],
  "complete": bool, "gaps": [RetrievalGap]
}

numeric_eligible is never trusted merely because the model sets it. The deterministic verifier recomputes eligibility, and document/chain/citation budgets are applied before verification and downstream use. Only verified chains are projected into the legacy Decision interface.

Resolved public outcome learning creates Retrieval Skills only as candidate records. After a mutation Child names a candidate ID in its desired eventual active_skill_ids, only that exact ID is projected through the real agent/verifier during exact internal Train shadow evaluation. The trusted evaluator runs a second pre-label harness replay with that candidate withheld. After the complete screen/fold batch has scored, it may append one accepted version to the candidate-specific library only when those replays show at least three tasks from two entities, exact-quote validity 1.0, non-worse sMAE and sRMSE with one strict gain, necessity, and no added catastrophe. The replay retains the omitted harness's actual final candidate pool, including any alternative contextual candidate; it is never synthesized from the primary run. Inherited accepted Skills remain available as context, while leave-one-out and promotion evidence are required only for candidate IDs named and used by the Child. One task remains a candidate, and a Child with any named-but-unpromoted ID cannot become the Train winner, reach Dev, or be published. The shared seed/Parent library is not aliased; Dev, Public, unknown stages, and frozen inference resolve active Skills only and never project a candidate.

Authenticated Retrieval evolution checkpoint schema v2 stores canonical Skill histories and active-record origins in a deduplicated snapshot table. Every completed evaluation binds its exact pre/post snapshot hashes, and every Genome fingerprint binds its current candidate-library snapshot in the same atomic checkpoint publication. Resume authenticates the checkpoint digest and epoch, then revalidates snapshot digests, exact history, active provenance, canonical promotion evidence against the retained with-Skill and actual omitted candidate pools. Each contextual forecast must also reproduce exactly from its evidence chain after that chain is reverified against the immutable task documents and the independently parsed Child Genome's named Skill IDs. Replays for the same task must agree on one primary execution: its numeric baseline score and canonical pool digest must equal the trusted task trace's coding-oracle metrics and baseline digest, while its fully projected primary pool's digest and score must equal that trace's contextual-pool digest and contextual-oracle metrics. Evaluator-computed gates/metrics and an exact candidate-to-accepted copy may then be checked. The authenticated ordered completed-batch sequence binds each pre-state cache key and pre/post snapshot. Once its complete transition chain and final candidate snapshot validate, resume consumes each matching completed record exactly once—including earlier unchanged batches and earlier promotions—without rerunning it. Within an authenticated host record, missing, mismatched, semantically inconsistent, Dev-derived, unbound, or duplicate state fails closed; schema-v1 checkpoints are incompatible and are not silently migrated.

The operator HMAC key and trusted host evaluator are the Retrieval checkpoint trust root. Resume does not independently attest a nondeterministic harness execution beyond that authenticated host record. A trusted operator holding the key and current external anchor may intentionally reissue or migrate coherent state; that administrative action is outside the untrusted-model/tamper threat model above. Without current operator authority, even a coherently rewritten execution record is rejected. Authority key and expected-anchor values are removed before any LLM subprocess is built.

Every Retrieval generation requests exactly one child in each immutable scope:

The host rejects any child that changes a field outside its scope. On Train, it screens exactly eight cases drawn from one or more complete entities by default; those screening entities are disjoint from every remaining Train fold, but the eight cases need not name eight distinct entities. It promotes at most two children, then evaluates survivors over the remaining entity-disjoint folds. Dev opens once, after the Train winner is fixed, and runs Parent and Child with persistence, writers, and evolvers disabled. Acceptance requires a strict contextual-oracle gain with no regression in final mean sMAE/sRMSE, P90/P95, exact-quote validity, invalid/catastrophic counts, or the configured retrieval-quality tolerances. Public Regression IDs are excluded from mutation, evaluation caches, checkpoints, prompts, and acceptance; Public and hidden inference run only from a frozen release and never learn or evolve.

Run the deterministic no-token end-to-end proof:

python -m pytest -q tests/test_retrieval_e2e.py
python -m pytest -q tests/test_retrieval_e2e.py::test_fake_two_stage_smoke

The formal real-LLM command is provided for a future authorized run; it has not been run for the results in this repository. The operator authority paths must be outside RUN_DIR. Supply the key with at least 32 shell characters (the CLI also requires at least 32 UTF-8 bytes) without putting its value in the command line, repository, logs, or run artifacts:

read -r -s -p 'Retrieval checkpoint authority key: ' RETRIEVAL_CHECKPOINT_AUTHORITY_KEY
printf '\n'
export RETRIEVAL_CHECKPOINT_AUTHORITY_KEY

TASKS_FILE=external/Dr-CiK/full-download/Dr-CiK_public/tasks \
SPLIT_FILE=splits/drcik_public_80_20_99_v1.json \
RUN_DIR=runs/retrieval_evolution/formal_80_20 \
AUTHORITY_PATH=../retrieval-operator-state/formal_80_20.json \
AUTHORITY_HEAD_PATH=../retrieval-operator-state/formal_80_20.head.json \
AUTHORITY_ANCHOR_PATH=../retrieval-operator-state/formal_80_20.anchors \
scripts/run_retrieval_evolution.sh

On a fresh invocation, the host atomically creates $AUTHORITY_ANCHOR_PATH/bootstrap.json before its first checkpoint transaction. An operator or control-plane process must retain that file's external_anchor value independently of both the run tree and anchor ledger. On every restart, restore that retained value and the same authority key before invoking the identical command:

# Run this from a separate trusted operator process as soon as bootstrap.json appears.
# Replace this destination with an independently protected control-plane path.
RETAINED_ANCHOR_FILE=/absolute/operator-control-plane/formal_80_20.anchor
umask 077
python - ../retrieval-operator-state/formal_80_20.anchors/bootstrap.json \
  > "$RETAINED_ANCHOR_FILE" <<'PY'
import json
import sys

with open(sys.argv[1], encoding="utf-8") as source:
    anchor = json.load(source)["external_anchor"]
if not isinstance(anchor, str) or ":" not in anchor:
    raise SystemExit("invalid bootstrap external_anchor")
print(anchor)
PY

# On resume, read the independently retained value; never reread the ledger.
read -r -s -p 'Retrieval checkpoint authority key: ' RETRIEVAL_CHECKPOINT_AUTHORITY_KEY
printf '\n'
export RETRIEVAL_CHECKPOINT_AUTHORITY_KEY
read -r RETRIEVAL_CHECKPOINT_AUTHORITY_EXPECTED < "$RETAINED_ANCHOR_FILE"
export RETRIEVAL_CHECKPOINT_AUTHORITY_EXPECTED
TASKS_FILE=external/Dr-CiK/full-download/Dr-CiK_public/tasks \
SPLIT_FILE=splits/drcik_public_80_20_99_v1.json \
RUN_DIR=runs/retrieval_evolution/formal_80_20 \
AUTHORITY_PATH=../retrieval-operator-state/formal_80_20.json \
AUTHORITY_HEAD_PATH=../retrieval-operator-state/formal_80_20.head.json \
AUTHORITY_ANCHOR_PATH=../retrieval-operator-state/formal_80_20.anchors \
scripts/run_retrieval_evolution.sh

The runner consumes and scrubs both authority environment variables before constructing any LLM subprocess and prints only the non-secret command. The current CLI uses the conservative empty Morphology provider; therefore the real CLI path safely skips Round 2 until a real accepted Numerical/Morphology assumption provider is integrated. That integration and the run manifest (implementation commit, seed release hash, model/effort, budgets, split/verifier/metric hashes, metric cap, and output directory) must be frozen before starting the real 80/20 experiment.

Baseline methods and current results

Method definitions

Full 199-task public-development results

These rows use the same public task set and the same Chronos backbone. Lower is better for all forecasting metrics. The scaled metrics are local development proxies, not official hidden-test scores.

Frozen 30-task Codex results

The earlier Codex Triad generated useful candidates on some tasks, but its Decision role and runtime were unreliable: the saved run records 683 Codex-stage calls and 252 stage failures. This negative result motivated citation-constrained selection, fallback rules, and the current evolution work.

Three-task development results

These runs use the three official public samples (task_42, task_163, and task_201) and are mechanism checks rather than unbiased benchmark results.

Additional saved diagnostics and missing comparisons

The main comparison for the current project is therefore:

Chronos
vs Codex Direct
vs Codex Contract
vs Codex Triad
vs the evolved three-agent harness

These numbers come from local public-development experiments, not the official Dr-CiK hidden-test leaderboard. Results from different task sets are not directly comparable. The new matched Prompt/Genome/Source evolution pilot is still running and is intentionally not reported here as a completed result. See docs/BASELINE_METHODS_AND_RESULTS.md for the complete audit and artifact-level details.

System flow

Experimental co-evolving three-agent loop

The image-inspired architecture is now executable as --system triad:

numbers only ──> Coding Agent ──> multiple executable forecast candidates
                                      │
documents ────> Retrieval Agent ──> verified evidence + typed impacts
                                      │
                                      v
                               Decision Agent
                         select / combine / ask again
                                      │
                                      v
                         probabilistic final forecast

delayed ground truth ──> coding coverage + retrieval quality + selection regret
                     ──> separate feedback for the three agents

The Coding Agent initially sees only numbers and generates backbone, transparent statistical, robust-history, and local-level hypotheses. The Retrieval Agent searches for evidence that distinguishes them and converts accepted prose into typed impacts. The Coding Agent can then generate evidence-conditioned candidates, while the Decision Agent selects or ensembles candidates and can request another retrieval round. Ground truth is used only after the future resolves, never during inference.

Candidate families do not receive hand-written model priors. Before forecasting the real horizon, each executable program is evaluated on up to three rolling historical cutoffs. Its base reliability is 1 / (1 + mean scaled validation MAE). Text evidence is a compatibility constraint rather than an arbitrary score bonus: a grounded active event can require an evidence-adjusted candidate, while a resolved event adds no numerical preference for any model family. If the history is too short to validate, the system conservatively preserves the configured backbone. The Decision Agent also avoids fixed score-margin ensembles; combining programs is deferred until out-of-fold stacking weights can demonstrate a validation gain.

Run the loop on a public Dr-CiK sample and write the delayed-feedback records:

python3 -m drcik_agent run-sample \
  --sample-dir external/Dr-CiK/sample \
  --system triad \
  --backbone chronos \
  --max-steps 3 \
  --top-k 5 \
  --learn-from-public-outcomes \
  --feedback-file outputs/triad/agent-feedback.jsonl \
  --evolution-file outputs/triad/evolution-policy.json \
  --output-dir outputs/triad

The current self-evolution layer is an interpretable online policy rather than neural-weight training. It updates learned candidate-family preferences, useful retrieval vocabulary, and decision-tag preferences after each resolved public task; these delayed learned preferences are logged separately from historical validation scores. The feedback log also exposes three distinct future training targets: candidate-set coverage for Coding, evidence quality for Retrieval, and selection regret for Decision.

Numbers-only Coding Agent and co-evolution

The executable Coding Agent now has an explicit information boundary and a nested evolution contract:

Input: historical timestamps + historical values only
  -> analyze statistical properties
  -> generate multiple falsifiable assumptions
  -> translate every assumption into executable Python
  -> sandbox execution
  -> rolling historical hindcast
  -> revise one failed assumption/program once
Output: assumption + failure condition + code + hindcast score

It never receives documents, retrieved evidence, gt_evidence, future values, or Retrieval Agent output. scripts/run_code_evolution.py executes this inner task-level loop. The returned program is selected by history-only validation rather than by an LLM-authored numerical answer.

The new outer loop in co_evolution.py evolves the reusable generation policy across resolved training tasks. It evaluates a population of versioned prompt bundles, attributes the weakest module from candidate coverage, evidence quality, and decision regret, then changes exactly one eligible prompt. The sequence is:

Coding assumptions + executable candidates
        -> Retrieval searches for evidence that tests those assumptions
        -> Decision cross-checks candidates against verified evidence
        -> final forecast
        -> delayed ground truth and labels score all three modules
        -> failure attribution chooses Coding OR Retrieval OR Decision
        -> mutate one prompt -> train/dev evaluation -> retain only validated bundles

Run the minimal population loop with scripts/run_co_evolution.py. An evolved JSON bundle can be loaded into the Codex triad with --agent-bundle. This is prompt- policy evolution, not neural-weight training and not graph-topology search.

All three reasoning roles can instead be backed by schema-constrained Codex calls:

.venv/bin/python -m drcik_agent run-sample \
  --sample-dir external/Dr-CiK/sample \
  --task-id task_42 \
  --system triad \
  --reasoning-agent codex \
  --backbone chronos \
  --chronos-model-id outputs/model-cache/chronos-bolt-small \
  --chronos-local-files-only \
  --codex-model gpt-5.6-sol \
  --codex-reasoning-effort high \
  --top-k 6 \
  --max-steps 2 \
  --output-dir outputs/task42-codex-triad

In this mode Codex proposes the executable numerical hypothesis families, autonomously searches the local corpus with exact-quote grounding, and selects among executed candidates. Python/Chronos still execute, backtest, and validate the numerical programs; Codex cannot emit or overwrite the final trajectory directly. Invalid citations, unknown candidate IDs, schema failures, and CLI failures are rejected or fall back visibly to the deterministic implementation. The exact prompts and Dr-CiK comparison are documented in docs/CODEX_TRIAD.md.

On the public task_42 mechanism test, the improved prompt found doc_1560, the Coding Agent materialized its “historical baseline and seasonality” claim as a backtested 22-step harmonic regime candidate, and the Decision Agent selected that executable candidate:

This is one public sample and demonstrates the mechanism only; it is not an aggregate claim.

history ──> diagnosis ──> Chronos-Bolt ──> immutable y_baseline
   │
documents ──> gap judge ──> next query ──> BM25 candidate pool
                                │                    │
                                │                    v
                                │        forecast-utility reranker
                                │                    │
                                │                    v
                                └──── insufficient ─ evidence grounding
                                                     │
                                                     v
                                       sentence-level Evidence State
                                                     │
                                          sufficient / missing gaps
                                                     │
                                                     v
                       importance compression ──> macro + micro outlook
                                                     │
                                                     v
                                      PostTime-style revise/preserve
                                                     │
                                                     v
                                     Last-Mile restricted workspace
                                                     │
                                                     v
                                    y_final + trajectories + audit trace

resolved training tasks ──> forecast-utility labels ──> frozen learned scorer
realized outcomes ──> post-hoc memory ──> later chronological tasks

The complete English architecture diagram, including the online inference path and offline learning/evolution path, is available in two standalone formats: docs/architecture.html and docs/architecture.svg.

The Chronos baseline is generated exactly once. Retrieval cannot rewrite historical values or y_baseline; it can only propose changes to y_final. This makes it possible to ask whether context improved the forecast instead of hiding the backbone and the contextual revision inside one opaque prompt.

Paper-derived design

The runtime is deliberately a synthesis rather than many complete frameworks stacked together:

See docs/PAPER_INTEGRATION.md for the module mapping and recommended ablations, and docs/CHRONOS_BACKBONE.md for the default backbone configuration. TimesFM remains available as an optional comparison; see docs/TIMESFM_BACKBONE.md.

Each loop iteration begins with a structured sufficiency decision. The controller selects the highest-value unresolved gap, constructs the next query, and can create new follow-up gaps from grounded evidence. For example, finding an anomaly dynamically creates a resolution_permanence gap; finding an unquantified future event creates an event_magnitude gap. The loop stops when gaps are resolved, the corpus is exhausted, progress stalls, or expected information gain falls below the configured cost threshold.

The verifier checks entity identity, temporal alignment, target relevance, whether the document answers the current question, and whether textual claims conflict with the observed numerical pattern. A document rejected only because it answers a different question remains available to later iterations.

Accepted evidence is translated into structured forecast impacts containing the event window, direction, permanence, forecast-horizon overlap, magnitude, confidence, and an auditable adjustment rule. The safe default does not directly apply generic textual effects such as increase by 20% or 2 times the usual demand: these magnitudes often refer to a different baseline or are already reflected in history. Only an explicit future timestamp-value pair or a history-backtested normal-regime projection may revise the numerical prior automatically. Other multiply/add candidates are preserved for an explicit unsafe ablation. An event that ended before the horizon produces a return_to_baseline instruction and is not extrapolated.

Each impact then becomes a proposal with an event type, affected range, action type, value, source documents, confidence, rationale, and any retrieved memory IDs. The workspace executor accepts only these actions:

Unsupported edits, out-of-horizon ranges, low-confidence changes, unsafe multipliers, and duplicate actions are rejected and recorded. Corroborating documents therefore do not multiply the same event effect twice.

To reproduce the deliberately unsafe generic-event ablation, add --allow-unvalidated-event-revisions to an iterative run.

Optional Codex reasoning agents

For a clean Codex-powered comparison, codex-direct is an intentionally unguarded full-corpus agent baseline. Chronos generates the same numerical prior used by our method; Codex then searches the task's local document directory and directly returns cited evidence and the complete final trajectory in one call. This baseline does not use our BM25 candidate stage, verifier, evidence-to-impact translator, memory, or restricted revision workspace:

python3 -m drcik_agent run-hf \
  --public-dev \
  --task-id task_117 \
  --system codex-direct \
  --backbone chronos \
  --chronos-model-id outputs/model-cache/chronos-bolt-small \
  --chronos-local-files-only \
  --codex-reasoning-effort high \
  --output-dir outputs/codex-direct-task117

The default reasoning effort for codex-direct is high, matching the style of the Codex configuration reported by Dr-CiK. The exact available Codex model depends on the installed CLI and account; use --codex-model to freeze it in a formal experiment. The output records Codex calls, cache hits, failures, latency, cited-document retrieval metrics, and forecast metrics. Invalid or failed Codex forecasts fall back visibly to the immutable Chronos baseline instead of silently using our hybrid agents.

A real task_42 smoke run of this baseline used one uncached high-effort Codex call (201.6 seconds). Codex cited four supporting documents and no distractors, but preserved Chronos exactly because it found no defensible numerical adjustment:

This is a single public smoke task, not an aggregate result. It illustrates the intended ablation: autonomous Codex found clean evidence, while the proposed evidence-to-impact and restricted revision path was still necessary to turn context into a numerical improvement.

codex-contract tests that bridge explicitly. Codex may search the full corpus and emit evidence plus a structured regime hypothesis, but it is forbidden to emit future numbers. For a grounded normal_seasonal contract, a numerical tool fits a trend-harmonic candidate, validates it on historical holdouts, blends it with Chronos according to validation gain, and applies the result through the restricted workspace:

python3 -m drcik_agent run-sample \
  --sample-dir external/Dr-CiK/sample \
  --task-id task_42 \
  --system codex-contract \
  --backbone chronos \
  --chronos-model-id outputs/model-cache/chronos-bolt-small \
  --chronos-local-files-only \
  --codex-reasoning-effort high \
  --output-dir outputs/codex-contract-task42

On task_42, Codex produced a 0.98-confidence normal_seasonal contract, correctly marked the software anomaly and promotion windows, and cited five supporting documents with no distractors. The history-only candidate achieved validation MAE 12.7622 versus 43.9295 for seasonal naive, resulting in a 0.709 blend weight. Final MAE fell from 72.7346 to 31.6472 (56.5% relative gain), matching the proposed safe hybrid on this task. The uncached Codex call took 121.4 seconds. This remains a single-task mechanism test rather than aggregate evidence.

Across all three official repository sample tasks, the contract system improved two tasks, preserved one, and harmed none. Explicit values are accepted only when a dated value in one grounded source is independently corroborated by another grounded source at the same local time and remains plausible under the observed numerical scale:

A frozen, label-free 30-task public-development evaluation is documented in docs/EVAL_30_RESULTS.md. It improves mean MAE from 162.6270 to 156.3621, with 2 improved tasks, 28 unchanged tasks, and no harmed tasks.

For task_163, eight of 24 future timestamps were present in a dated table and repeated in a second independently formatted report. Blending those corroborated anchors at weight 0.75 reduced MAE by 10.5%. For task_201, Codex still cited two time-series distractors, but their unanchored single-source numeric range supplied no corroborated timestamp-value pairs; all explicit revisions were rejected and Chronos was preserved.

Mean MAE fell from 27.7416 to 13.6800 (50.7%), and mean RMSE fell from 39.4859 to 22.5559 (42.9%). The three-task result remains a mechanism test, not evidence that every task or contract type improves.

The deterministic text modules remain the default for reproducibility. A logged-in local Codex CLI can replace query planning, semantic evidence verification, and evidence-to-impact translation while leaving Chronos and the restricted forecast workspace unchanged:

drcik-agent run-sample \
  --sample-dir external/Dr-CiK/sample \
  --task-id task_42 \
  --backbone chronos \
  --chronos-model-id outputs/model-cache/chronos-bolt-small \
  --chronos-local-files-only \
  --reasoning-agent codex \
  --codex-stages query,verify,impact \
  --codex-reasoning-effort low \
  --codex-cache-dir outputs/codex-cache

Every Codex call uses an ephemeral read-only sandbox, a strict JSON output schema, exact-quote grounding, source-ID validation, and an on-disk content-addressed cache. If the CLI times out, is unavailable, or returns invalid JSON, that stage falls back to the deterministic agent. Codex never receives future_values, gt_evidence, document role labels, or document subtype labels, and it cannot execute forecast revisions directly. Verifier-selected exact quotes are pinned through context compression. Daily timestamp-value blocks are parsed only inside these verified quote boundaries, preventing superficially similar distractor tables from changing the forecast.

For lower-cost experiments, use only the semantic verifier:

--reasoning-agent codex --codex-stages verify

On the public task_42 smoke test, the two-step full Codex loop matched the safe rule loop: MAE decreased from the Chronos baseline of 72.7346 to 31.6472, with no harmful revision. Codex increased retrieval precision from 0.8333 to 1.0000 on this single task but reduced the supporting-document recall proxy from 0.3846 to 0.2308. Six uncached Codex calls took 188.9 seconds.

An exploratory oracle bottleneck scan then identified tasks where the safe rule loop preserved Chronos but correct retrieved context could improve the downstream forecast. On two such hard cases, verifier-only Codex recovered the exact future-value blocks and reached the oracle ceiling:

The matched current rule loop remained unchanged on both tasks, while Codex produced no harmful revision. These cases were selected using public oracle diagnostics, so they demonstrate mechanism and implementation—not an unbiased aggregate accuracy estimate. The unchanged default rule system was rerun on all 199 public tasks after adding the Codex-only parsing gate and retained its prior 7-improved / 0-harmed safety result.

In the normal inference path, Dr-CiK's role, subtype, future_values, and gt_evidence fields are never exposed to the loop. Public labels are used only after a run to calculate development metrics. The explicitly separate --oracle-evidence diagnostic described below is the sole exception: it bypasses retrieval on public tasks to measure the downstream evidence-to-forecast ceiling and is rejected for hidden-test inference.

Post-hoc memory follows the same separation. A run never reads its own future values. Only after an outcome is explicitly recorded can the system compare y_baseline, y_final, and the actual series, store whether a revision helped, and use that lesson as a shrinkage prior for later matching events.

Quick start

Clone Dr-CiK and run its three official sample tasks:

git clone https://github.com/ServiceNow/Dr-CiK.git external/Dr-CiK
python3 -m venv .venv
source .venv/bin/activate
pip install -e '.[chronos]'

drcik-agent run-sample \
  --sample-dir external/Dr-CiK/sample \
  --output-dir outputs/agent-loop

The project entrypoint can run directly from source after installing the Chronos runtime:

pip install 'chronos-forecasting>=2.2.0'
python3 -m drcik_agent run-sample \
  --sample-dir external/Dr-CiK/sample \
  --output-dir outputs/agent-loop

The package and the amazon/chronos-bolt-small checkpoint are loaded lazily on the first forecast. The checkpoint is downloaded from Hugging Face unless it is already cached.

Useful loop controls:

drcik-agent run-sample \
  --sample-dir external/Dr-CiK/sample \
  --output-dir outputs/agent-loop \
  --system iterative \
  --max-steps 10 \
  --top-k 5 \
  --max-no-progress 4 \
  --convergence-tolerance 0.002 \
  --candidate-multiplier 3 \
  --context-character-budget 12000 \
  --min-information-gain 0.05 \
  --revision-threshold 0.60

Chronos controls:

drcik-agent run-sample \
  --sample-dir external/Dr-CiK/sample \
  --backbone chronos \
  --chronos-model-id amazon/chronos-bolt-small \
  --chronos-device-map cpu \
  --chronos-max-context 2048 \
  --chronos-max-horizon 1024 \
  --chronos-cache-dir outputs/model-cache

Chronos is the default and failure is explicit. The system does not silently switch to the old statistical model. For an intentional degraded-mode run, add --allow-statistical-fallback; the recorded baseline method will begin with statistical_fallback:. For a clean ablation, use --backbone statistical.

--top-k 5 --candidate-multiplier 3 retrieves 15 candidates, scores all 15 for forecasting utility, and sends only the best 5 to the verifier. The ranking and compression modules never see future_values.

Optional outcome memory for sequential research runs:

drcik-agent run-sample \
  --sample-dir external/Dr-CiK/sample \
  --output-dir outputs/with-memory \
  --memory-file outputs/forecast-memory.jsonl \
  --learn-from-public-outcomes

--learn-from-public-outcomes is intentionally opt-in and is rejected for the hidden test split. For a clean benchmark comparison, keep it off and record outcomes only in a separate, chronologically valid backtest.

--top-k is the number of new documents inspected at each iteration. The original one-pass baseline is preserved for ablations:

drcik-agent run-sample \
  --sample-dir external/Dr-CiK/sample \
  --output-dir outputs/one-pass \
  --system one-pass \
  --top-k 8

Outputs

• forecasts.jsonl: Dr-CiK forecasting submission format with 100 trajectories per task.
• deep_research.jsonl: accepted document IDs and extracted evidence.
• loop_trace.jsonl: every query, candidate, verifier verdict, structured evidence utility score, compression decision, macro/micro outlook, revision decision, accepted/rejected action, belief update, forecast summary, and stop decision.
• run_report.jsonl: per-task diagnosis, belief state, development metrics, and the full forecast workspace containing historical observations, immutable y_baseline, editable y_final, proposals, action results, and memory references.
• summary.json: aggregate development metrics.

Local sMAE, sRMSE, and sCRPS values are explicitly development proxies. Official hidden-test scores are calculated by the Dr-CiK maintainers. When a workspace is used, the report also includes baseline_mae, revision_value_mae, relative_revision_gain, and harmful_revision to measure whether the last-mile agent actually improved the forecasting backbone. It also reports revision_accept_rate, revision_fallback_rate, mean_predicted_revision_utility, context_retention_ratio, mean_belief_sufficiency, gap_coverage, mean_expected_information_gain, retrieval_turns, and documents_inspected.

Public development split

pip install -e '.[chronos,huggingface]'
drcik-agent run-hf --public-dev --output-dir outputs/public-dev

Use --limit 5 for a short development run. --hidden-test creates submission files without local forecast scores because the hidden labels are unavailable.

Oracle-evidence bottleneck diagnostic

Use this only on public labeled development tasks. It replaces retrieved documents with the task's gt_evidence annotations while keeping the same backbone, impact translator, revision gate, workspace actions, and uncertainty sampler:

drcik-agent run-sample \
  --sample-dir external/Dr-CiK/sample \
  --output-dir outputs/chronos-oracle-evidence \
  --oracle-evidence

This is not a valid test-time system and is not an official Dr-CiK result. The CLI and runtime both reject it when labels are hidden. Its purpose is causal debugging: if oracle evidence helps but normal retrieval does not, retrieval is a bottleneck; if even oracle evidence does not help, evidence-to-impact translation or revision is also a bottleneck.

With Chronos-Bolt Small, seed 7, and the three official repository samples, the current controlled diagnostic is:

Two changes close the oracle gap on these three samples. First, the context-point parser now reads wide Markdown forecast tables, allowing the normal loop to recover eight explicit future irradiance points for task_163. Second, a regime-normalization reviser translates verified "return to normal" evidence into a numerical trajectory: it fits a trend-plus-harmonic model to the most recent two observed cycles, validates it on the last cycle, and blends it with Chronos only when this history-only backtest beats seasonal naive. On task_42, its 22-step model has validation MAE 12.76 versus 43.93 for seasonal naive and reduces future MAE by 56.5%. No future values or benchmark role labels enter either decision.

Full public-split safety evaluation

We then evaluated all 199 labeled public synthetic tasks with the same Chronos-Bolt Small backbone. Hidden human tasks were not used. The sMAE, sRMSE, and sCRPS figures below are local proxies produced by this repository, not official leaderboard scores.

The safe loop changed only 7 of 199 forecasts (3.5%): all seven changes improved MAE and none harmed it. Five were history-backtested normal-regime projections and two were explicit future timestamp-value overrides. This result is why generic textual multiply/add actions are now opt-in rather than the default.

Tests

python3 -m unittest discover -s tests -v

Current scope

The first controlled comparison should be backbone only vs. oracle context vs. one-pass retrieval vs. iterative retrieval + unrestricted revision vs. iterative retrieval + restricted workspace revision, all using the same backbone. Report both forecast accuracy and revision value (baseline error - final error) so retrieval gains are separated from backbone quality and harmful context edits.