ALFA-updated

ALFA-updated is a repaired execution environment and grader for the 300 public requests in InterCode-ALFA. It preserves task requirements and submitted model commands, while correcting documented transport, reset, fixture, reference and output-checking defects. Its scores must be identified as ALFA-updated, separately from original ALFA.

The complete change log lists every repair and its reason. REASONING.md explains the correctness boundaries, and known limitations states remaining weaknesses.

Install and prepare

The pinned release assets support Linux x86_64, Docker and Python 3.11+. Docker must be running and accessible to your user. The embedding service runs on CPU. It is an embedding comparator, not a new generative LLM judge.

git clone https://github.com/dirac-run/ALFA-updated.git
cd ALFA-updated
python3 -m venv .venv
.venv/bin/pip install -r requirements.lock
.venv/bin/pip install --no-build-isolation .
.venv/bin/alfa-prepare

From a GitHub checkout, preparation derives the asset download URL from its origin remote and the single release tag v0.1.0. It downloads only missing assets, verifies SHA256 checksums, loads the exact five Docker image IDs, and prepares the locked embedding runtime/model. It never rebuilds fixtures, updates the lock, or pulls a mutable model tag. The release downloads contain all seven archives and their checksum file.

For a source archive or installed wheel without Git metadata, pass the asset base URL with alfa-prepare --release-url URL, or download the release's seven .tar files into one directory and use alfa-prepare --asset-dir /path/to/assets. Cache files go under ~/.cache/alfa-updated/0.1.0; ALFA_CACHE_DIR can choose another directory.

Start the embedding service in a separate terminal:

.venv/bin/alfa-embedding

This launches the exact locked Ollama runtime and mxbai-embed-large model on localhost. It refuses occupied ports and does not replace other services. For another port, use alfa-embedding --port 11435 and set ALFA_EMBEDDING_URL=http://127.0.0.1:11435 when grading. This changes only the service address, not its model or scoring parameters.

Grade a model

Generate a command for each task using your model's own prompt/inference setup. Submit each command without repairing or rewriting it:

from icalfa import submit_command, close

try:
    score = submit_command(index=0, command="ls")
finally:
    close()

The compatible import remains icalfa; install this package in a separate environment from upstream icalfa. Task indices are zero-based, 0 through 299. Public task requests are in data/tasks.jsonl.

For a full batch, save exactly 300 unique rows in JSONL form:

{"index":0,"command":"ls"}

.venv/bin/alfa-grade --predictions commands.jsonl --output results/model-run

The output directory must be new. A run freezes commands and grader inputs, records per-task execution/checking evidence, and refuses changes to its definition during grading. Each fixture uses one owned container, resetting state between candidate/reference executions and subsequent tasks. The evaluator removes its containers and network on close.

Scoring and fairness

Output scoring is a documented hybrid: supported task-information contracts check correctness directly; remaining cases retain the locked embedding model, the original 1000-character prefix and strict similarity threshold > 0.75. Filesystem tasks check actual types, contents, permissions and links. No new generative LLM judging is added.

Repairs are applied equally to all models. Candidate outputs, requested scope, ordering and destructive effects are not relaxed to favor a model. Fixture and reference changes are explicit in data/fixture_errata.jsonl and data/reference_errata.jsonl. Positive and deliberately incorrect controls are included in tests/.

Report passes / 300 together with model failures and benchmark errors. Infrastructure errors have a null score and are disclosed; they do not justify shrinking the denominator. Live host/network observations can vary, and a passing embedding comparison is not proof of universal semantic correctness. Report the model, quantization, prompt, decoding settings and benchmark definition recorded in the run manifest.

Complete repair inventory

Verify the implementation

With prepared assets and the embedding service running:

.venv/bin/pytest -q -p no:cacheprovider tests

The lock pins task/errata/rule files, dependencies, prepared Docker images, embedding runtime and model. Docker build recipes are included for inspection; rebuilding with today's package repositories does not reproduce the locked image IDs. Use the accompanying verified archives for this release.

Credit and license

Based on westenfelder/InterCode-ALFA commit 2d3a69473a68569828ab0b4859073ef4a0ae482c, derived from Princeton NLP's InterCode. Canonical requests come from westenfelder/NL2SH-ALFA, revision a99cb5784cf5c2a42b1cc26c1903d9c3b35206ba.

The upstream MIT license is retained in LICENSE.txt. Embedding/runtime archives retain their own license notices. The repaired public API is submit_command or Evaluator; unsupported historical Gym and external judging interfaces are not part of this package.