Agent Distillation Local Runner

This folder now has a local, non-Modal workflow for reproducing the agent-distillation evaluations and using the fine-tuned checkpoints for inference.

The old modal_app_clean.py and README (1).md are still useful as historical Modal references. Use these new files for local work:

Requirements

• Linux machine with an NVIDIA GPU for vLLM serving.
• Python 3.10+.
• Git.
• Hugging Face access for the Qwen and adapter checkpoints.
• opencode CLI. Confirm with:

opencode --version

The opencode provider config uses the current OpenAI-compatible provider pattern documented by opencode, with @ai-sdk/openai-compatible and options.baseURL. vLLM serves /v1/chat/completions, /v1/models, and other OpenAI-compatible endpoints.

1. Create an environment

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements-local.txt

If vllm install fails on macOS, move this workflow to a Linux/NVIDIA CUDA environment. The notebook can open anywhere, but serving/evaluation needs GPU memory.

2. Clone and patch the research repo

python agent_distillation_local.py setup --install

This clones https://github.com/Nardien/agent-distillation.git into .agent_distillation/agent-distillation, patches the dependency conflict from the Modal script, and installs the research package.

3. Merge a fine-tuned checkpoint

base does not need a merge. The fine-tuned variants do:

python agent_distillation_local.py merge --student-key qwen25_1_5b --variant distill
python agent_distillation_local.py merge --student-key qwen25_1_5b --variant ftp

Supported student keys:

qwen25_0_5b
qwen25_1_5b
qwen25_3b

Supported variants:

base
distill
ftp
distill_sag
ftp_sag

Merged checkpoints are saved under .agent_distillation/cache/merged_models/.

4. Run an evaluation

Start with a small subset:

python agent_distillation_local.py eval \
  --student-key qwen25_1_5b \
  --dataset-key gsm_hard \
  --variant base \
  --limit 10 \
  --max-model-len 8192 \
  --max-tokens 1024 \
  --max-steps 5

Run a merged model:

python agent_distillation_local.py eval \
  --student-key qwen25_1_5b \
  --dataset-key aime \
  --variant distill \
  --limit 0 \
  --max-model-len 16384

Results go to .agent_distillation/cache/results/.

Useful result commands:

python agent_distillation_local.py list-results
python agent_distillation_local.py show-score --result-name qwen25_1_5b_base_gsm_hard_10
python agent_distillation_local.py summarize --student-key qwen25_1_5b --variant base --limit 10

5. Run inference through the opencode dashboard

The inference path uses vLLM for model serving and opencode serve for the backend/dashboard. Register the selected model with the live opencode backend before testing it in the dashboard.

Start the backend:

export VLLM_API_KEY=token-abc
python inference_opencode.py serve \
  --student-key qwen25_1_5b \
  --variant distill \
  --vllm-port 8000 \
  --opencode-port 4096

This command:

1. Resolves the merged checkpoint.
2. Starts vllm serve on http://127.0.0.1:8000/v1.
3. Writes/updates opencode.json.
4. Starts opencode serve on http://127.0.0.1:4096.
5. Registers vllm/qwen25_1_5b_distill with the running opencode backend.

Open the dashboard URL printed by the command and select/use the registered vllm/... model there.

If opencode serve is already running, register the model against that live backend:

export VLLM_API_KEY=token-abc
python inference_opencode.py register \
  --student-key qwen25_1_5b \
  --variant distill

To smoke-test dashboard selection, create a dashboard session through the same backend API that the web UI uses:

python inference_opencode.py dashboard-test \
  --student-key qwen25_1_5b \
  --variant distill \
  --create-session-only

When vLLM is running, omit --create-session-only to send a real dashboard/API prompt:

python inference_opencode.py dashboard-test \
  --student-key qwen25_1_5b \
  --variant distill \
  --prompt "Solve: If 3x + 4 = 19, what is x?"

Register any Hugging Face or local vLLM model

Use the generic helper instead of editing opencode.json by hand. For the current local agent model served through the SSH tunnel on port 11404, run:

python register_opencode_model.py agent-distillation/agent_distilled_Qwen2.5-1.5B-Instruct \
  --served-name agent \
  --llm-port 11404 \
  --context-limit 32768 \
  --output-limit 8192

For a full fine-tuned Hugging Face model repo such as honghak/qwen2.5-0.5-tool_call_sft, run:

python register_opencode_model.py honghak/qwen2.5-0.5-tool_call_sft \
  --llm-port 8000 \
  --context-limit 8192 \
  --output-limit 2048 \
  --create-session

The helper:

1. Derives a served name, for example qwen2_5_0_5_tool_call_sft.
2. Writes/updates opencode.json.
3. Adds the local math-calculator MCP server to opencode.json.
4. Adds a project math agent and makes it the default agent.
5. Uses AGENTS.md/opencode_math_tool_instructions.md so the agent is told to call math-calculator_calculate for arithmetic.
6. Registers the model with the running opencode serve backend when it is reachable at http://127.0.0.1:4096.
7. Prints the matching vllm serve command and optional dashboard session URL.

Start vLLM with the printed command. For the Hugging Face example above:

export VLLM_API_KEY=token-abc
vllm serve honghak/qwen2.5-0.5-tool_call_sft \
  --host 127.0.0.1 \
  --port 8000 \
  --served-model-name qwen2_5_0_5_tool_call_sft \
  --max-model-len 8192 \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --api-key "$VLLM_API_KEY"

Because opencode.json includes the math-calculator MCP server, OpenCode sends tools to vLLM with tool_choice: "auto". vLLM must be started with --enable-auto-tool-choice and a matching --tool-call-parser, otherwise the dashboard returns "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set.

Math calculator MCP

The generated opencode.json includes:

"mcp": {
  "math-calculator": {
    "type": "local",
    "command": ["python", "math_calculator_mcp.py"],
    "enabled": true
  }
}

OpenCode starts this local MCP server and exposes a calculate tool for exact arithmetic checks. The tool supports arithmetic operators, parentheses, common functions like sqrt, sin, log, and constants pi, e, and tau.

The generated config also sets a project math agent as the default agent. This keeps the prompt small and allows only math-calculator_calculate, which is important for small local Qwen models. In the dashboard, start a new session or select the Math agent before testing calculator calls.

6. Notebook workflow

Open:

jupyter notebook agent_distillation_local.ipynb

The notebook is intentionally thin: it explains the flow and calls the functions in agent_distillation_local.py, instead of hiding another long script in notebook cells.

Common issues

Merged model not found

Run the matching merge command first:

python agent_distillation_local.py merge --student-key qwen25_1_5b --variant distill

vLLM server did not become ready

Check the vLLM log in .agent_distillation/cache/results/. Common causes are insufficient GPU memory, a too-large --max-model-len, or missing Hugging Face access.

opencode cannot find model vllm/...

Make sure opencode serve is running, then register the model with that live backend:

python inference_opencode.py register \
  --student-key qwen25_1_5b \
  --variant distill

Then refresh the dashboard and use the vllm/... model shown in the model selector.