llvm-hackme

A replacement for llvm-mutation-based-fuzz-service. It monitors open pull requests to llvm/llvm-project, performs fuzzing on proposed middle-end patches, and reports bugs found (opt crashes or Alive2 miscompilations) as PR review comments.

Motivation

LLVM receives hundreds of middle-end patches each week. Reviewer bandwidth is limited, and subtle correctness bugs often survive code review. This service automates the most tedious part of correctness verification -- mutation-based fuzzing -- so that reviewers can focus on high-level design decisions while the bot catches regressions mechanically.

The service checks PRs that touch passes such as InstCombine, InstSimplify, GVN, EarlyCSE, SCCP, Reassociate, SimplifyCFG, ConstraintElimination, VectorCombine, AggressiveInstCombine, CorrelatedValuePropagation, and PhaseOrdering, as well as shared analysis infrastructure: KnownBits, KnownFPClass, ValueTracking, ConstantFolding, and InstructionSimplify.

For deeper review of individual PRs, see Archer.

Configuration

The following environment variables are required:

Optional variables (with defaults):

Quick Start

# 1. Set environment variables
export GITHUB_TOKEN=ghp_...
export OPENAI_ENDPOINT=https://api.openai.com/v1
export OPENAI_AUTH_KEY=sk-...
export OPENAI_MODEL=gpt-4o-mini

# 2. Install dependencies
uv sync

# 3. Run (TUI mode by default)
uv run python main.py

# Or headless mode
uv run python main.py --plain

The first run will clone llvm/llvm-project and alive2, then build the LLVM toolchain (opt, llvm-extract, llvm-reduce, alive-tv, fuzz tools). Subsequent runs only rebuild when the baseline moves forward.

How It Works

1. Scan -- polls GitHub for open, non-draft PRs targeting main (excluding reverts). Checks if the PR touches relevant middle-end files.
2. LLM Review -- a lightweight OpenAI call classifies each patch chunk as malicious or innocuous before any build or execution.
3. Build -- prepares a clean LLVM worktree at the PR head, builds opt, and assembles the full toolchain (baseline + PR opt, alive-tv, mutation tools).
4. Fuzz — when the patch touches test files, extracts seed functions from the changed .ll tests, mutates them, and runs the PR opt with the guessed pipeline. Alive2 checks correctness.
5. Hack — if fuzzing finds nothing (or if the patch is source-only and fuzz is skipped), a lightweight LLM agent (openCode headless) analyzes the patch, reads the LLVM source, and attempts to construct a targeted test case. Time budget is configurable via LLVM_HACKME_HACK_BUDGET_SECONDS (default 20 min).
6. Verify -- each suspected bug is regression-tested against the baseline opt to confirm it is a new issue.
7. Report -- posts a GitHub issue comment with the IR reproducer (crash stacktrace or Alive2 counterexample) and requests changes on the PR.

Future Scope

• Expand coverage to additional LLVM middle-end passes and analyses.
• Tighten integration with the llvm-autofix pipeline so that confirmed bugs can be automatically narrowed down to a minimal test case and submitted to the LLVM issue tracker.

License

Apache-2.0 -- see the LICENSE file.