O11y Bench Leaderboard Submissions

This repository accepts leaderboard submissions for o11y-bench, Grafana's benchmark for LLM agents on observability and SRE tasks.

How to Submit

1. Fork this repository
2. Create a new branch for your submission
3. Add your submission under submissions/o11y-bench/1.0/<agent>__<model(s)>/
4. Open a Pull Request

Submission Structure

submissions/
  o11y-bench/
    1.0/
      <agent>__<model>/
        metadata.yaml       # Required: agent and model info
        <job-folder>/       # One or more Harbor job directories
          config.json
          result.json
          <trial-1>/result.json
          <trial-2>/result.json
          ...

Required: metadata.yaml

Each submission must include a metadata.yaml file with the following fields:

agent_url: https://...                # Required: link to agent repo/docs
agent_display_name: "My Agent"        # Required: display name for leaderboard
agent_org_display_name: "My Org"      # Required: organization name

models:                                   # Required: list of models used
  - model_name: openai/gpt-5.4-mini       # Required: model identifier
    model_provider: openai                # Required: provider (openai, anthropic, google, etc.)
    model_display_name: "GPT-5.4 Mini"    # Required
    model_org_display_name: "OpenAI"      # Required
  # - Other models if your agent used multiple

Job Directory Requirements

Each job directory must contain all of the contents of the Harbor run you want scored, including:

• config.json
• result.json
• all trial subdirectories
• agent logs and artifacts downloaded with the run
• verifier output for each trial

Validation Rules

Submissions are expected to preserve the benchmark's shipped evaluation settings. In the current local o11y-bench repo:

• Harbor runs use timeout_multiplier = 1.0; submissions should not override it
• Every task currently ships with agent.timeout_sec = 600.0
• Every task currently ships with verifier.timeout_sec = 300.0
• Every task currently ships with environment.build_timeout_sec = 600.0
• No resource overrides (override_cpus, override_memory_mb, override_storage_mb)
• All trial directories must have valid result.json files
• Trial directories must include the rest of the run artifacts, not just copied result files
• The current benchmark runner defaults to n_attempts = 3 per task and n_concurrent = 8

If leaderboard policy later requires a different minimum number of attempts per task, document that policy here and in the validator rather than relying on stale benchmark README text.

Recommended Run Setup

O11y Bench runs on Harbor. A typical benchmark run looks like:

cd o11y-bench
mise run bench:job -- --model openai/gpt-5.4-mini

This writes a Harbor job directory under jobs/. By default, the current runner executes 3 attempts per task and uses concurrency 8.

Copy the completed job directory into your submission folder and add a metadata.yaml file alongside it.

If you want an explicit attempt count in the recorded run, pass it through the benchmark CLI:

mise run bench:job -- --model openai/gpt-5.4-mini --n-attempts 3

Example Submission Path

submissions/o11y-bench/1.0/MyAgent__GPT-5.4-Mini/
  metadata.yaml
  2026-04-10__14-30-00/
    config.json
    result.json
    ...

Submission Process

1. Open a Pull Request with your submission
2. Fix any validation issues reported by CI or maintainers
3. After approval and merge, the submission can be imported into the leaderboard

Questions?

Open an issue in this repository.