🔨 Carpenter

An LLM-powered GitHub bot for issue triage automation in the Nuxt ecosystem

Carpenter is an automated assistant that helps with issue triage in the Nuxt GitHub repository. It uses AI to analyse issues, categorise them, detect missing reproductions, handle reopened issues, and translate non-English content.

What it does

On issues.opened:

• Categorises the issue (bug / enhancement / documentation / spam) and sets the GitHub issue type
• Labels bugs without a reproduction as needs reproduction, vague reports as needs details, and reports with a minimal StackBlitz/CodeSandbox reproduction as ✨ good reproduction
• Labels possible regressions and Nitro server engine issues
• Labels bugs with the major version they are reported against (3.x, 4.x, 5.x), based on the Nuxt version in the environment section
• Labels the affected area (pages, components, layers, kit, types, ...), non-default bundler (bundler:webpack, bundler:rspack) and platform (platform:windows, platform:bun) when the issue is specific to them
• Adds pending triage when nothing else applies
• Transfers spam issues to a separate repository (falling back to a spam label if transfer fails)
• Translates non-English issues (title prefix + appended body translation)

On issues.edited and issue_comment.created:

• Removes needs reproduction when a reproduction is added, reopening the issue if needed
• For comments on closed issues, runs a deeper analysis (recent comments + timeline) to decide whether to reopen: regressions, "not planned" issues with new evidence, or duplicates that turn out to be distinct. Guarded by confidence levels and reopen history. Comments that confirm a fix, thank someone or ask about future plans never reopen, and short "thanks, that works now" comments skip the analysis entirely.

On issues.labeled:

• spam label added: transfers the issue to the spam repository

Comments and edits from bots and repository collaborators are ignored; humans always have the final say.

Label-triggered comments (e.g. reproduction guidance when needs reproduction is added, AI contribution policy when possible bot is added) remain as plain GitHub Actions workflows in nuxt/.github; they need no LLM, and labels added by Carpenter trigger them normally since Carpenter acts with its own identity rather than the Actions GITHUB_TOKEN.

🛠️ Tech Stack

• Built on Nuxt with Nitro server routes
• AI SDK with the Vercel AI Gateway for model access
• Valibot schemas for structured model output
• GitHub API via Octokit
• Deployed on Vercel

🚀 Getting Started

Local Development

corepack enable
pnpm install
pnpm dev

In dev mode, GitHub API calls are logged rather than executed, and webhook signature validation is skipped.

Environment Setup

Create a .env file with:

# Vercel AI Gateway (provisioned automatically via OIDC when deployed on Vercel)
AI_GATEWAY_API_KEY=<your-ai-gateway-api-key>

NUXT_GITHUB_TOKEN=<your-github-token>
NUXT_GITHUB_TARGET_REPOSITORY_NODE_ID=<node id of repo to transfer spam issues to>
NUXT_WEBHOOK_GITHUB_SECRET_KEY=<your-webhook-secret-key>

# Optional model overrides (any Vercel AI Gateway model identifier)
NUXT_AI_SIMPLE_MODEL=openai/gpt-4o-mini
NUXT_AI_COMPLEX_MODEL=openai/gpt-4o

Other optional overrides: NUXT_TRIAGE_PROJECT_NAME, NUXT_TRIAGE_TRANSLATE_ISSUES, NUXT_TRIAGE_MAIN_BRANCH_MAJOR (the major currently developed on main, used when a report only says "nightly" or "main" without a version number).

Evaluating the classifier

Prompt or model changes can be checked against historical issues and their human-applied labels:

pnpm eval:fetch   # pulls ~200 triaged issues into eval/issues.json (uses gh auth or GITHUB_TOKEN)
pnpm eval         # runs the new-issue analysis and prints per-label precision/recall

Model responses are cached per model in eval/; pass --fresh to re-run the model and --verbose to list every mismatch. EVAL_MODEL, EVAL_LIMIT, EVAL_CONCURRENCY and EVAL_REPO override the defaults. Labels that humans rarely apply (version labels, ✨ good reproduction) will show low precision simply because the ground truth is missing, so read the numbers alongside the --verbose output.

GitHub webhook

Configure a webhook on the repository you want to monitor:

• Payload URL: https://<your-deployment>/api/webhook
• Content type: application/json
• Secret: <your-webhook-secret>
• Events: Issues, Issue comments

📄 License

Published under MIT License.