agent-skills

Installable QA skills for teams that want more confidence when shipping bug fixes and features.

qa-strategist gives your coding agent the behavior of a strong QA analyst: clarify expected behavior, identify the highest-risk paths, and turn that into concrete tests, release checks, and safer rollout decisions.

It is built for teams that do not just want advice. They want stronger evidence before shipping.

Who This Is For

• Teams shipping bug fixes or features without enough QA confidence
• Developers who want clear, actionable guidance instead of generic testing advice
• Projects that need a portable skill they can install quickly and use across agents

What Confidence Looks Like

When you use qa-strategist, you should expect it to:

• ask a few targeted questions before recommending a large plan
• focus on the user flows most likely to break or hurt
• turn expected behavior into concrete tests, checks, and rollout safeguards
• push bug-fix work toward reproduction, a failing test, a minimal fix, and verification
• give a clear go/no-go view when you ask about feature or release readiness

Why Tests Matter Here

This skill treats tests as the strongest confidence signal available when shipping software:

• tests make expected behavior explicit
• tests turn bug fixes into verified regressions that stay fixed
• tests give coding agents and humans the same definition of done
• tests reduce "it looks fine" decisions before release

Quick Install

Fastest path:

npx skills add cpoisson/agent-skills

If you do not want to use the installer, copy qa-strategist/ into .agents/skills/qa-strategist/.

Start Using It

Example prompts:

• Assess the main QA risks in this repo
• What should we test before shipping this bug fix?
• Help me reproduce this bug and write the first failing test
• Turn this acceptance criteria into a test plan
• Can we start this feature, or are we missing too much?
• What are the biggest release risks right now?

Skill Catalog

What Is Included

Evaluation Metrics

This repository keeps the skill measurable with a deterministic critic and a semantic trigger-plus-routing judge.

• Deterministic quality gate:
  • total quality score at least 90
  • overall trigger and routing score at least 90
  • portability score at least 90
  • description score at least 85
  • at least 2 eval cases per routed sub-skill
  • single-route load budget at most 2400 words
  • two-route load budget at most 3600 words
• Semantic routing gate:
  • minimum overall trigger and routing accuracy 0.85

The eval suite now includes:

• positive route-selection cases
• guided and ambiguous prompts
• negative controls that should not trigger the skill
• reporting split across trigger accuracy, triggered-route accuracy, and overall accuracy

Run the checks with:

python3 scripts/skill_critic.py --thresholds-file evals/skill_critic_thresholds.json --markdown-out EVALUATIONS.md
python3 scripts/semantic_skill_judge.py --provider ollama --model qwen2.5:1.5b --thresholds-file evals/semantic_judge_thresholds.json --markdown-out /tmp/semantic-judge.md

The latest deterministic report is published in EVALUATIONS.md.

Agent And Skill Model

The shared skill files are portable and procedure-focused. The optional agent files add a stronger persona and working style on top.

• Use the skill alone when you want lightweight routing and reusable QA workflows
• Use the agent template when you want a more opinionated QA strategist persona

Contributing

Each skill should:

• address one clear QA domain
• stay assistant-neutral
• include templates, checklists, decision trees, or test-oriented artifacts
• stay concise enough to load cheaply

Open an issue before adding a new skill.

Agent-Specific Notes

Codex

• Install the shared skill into .agents/skills/qa-strategist/
• Codex-specific repository guidance belongs in your project-level AGENTS.md
• Use the shared skill for routing and keep Codex behavior hints in the adapter layer

Claude

• Install the shared skill into .agents/skills/qa-strategist/
• If you want a Claude-specific agent entrypoint, copy .claude/agents/qa-strategist.md into your project
• Start with direct prompts such as what should we test before shipping this fix?

GitHub Copilot

• Install the shared skill into .agents/skills/qa-strategist/
• Add repo-level Copilot guidance with .github/copilot-instructions.md if you want this repo's adapter behavior
• If you want a stronger persona, also copy agents/qa-strategist.agent.md into your project

Related Work

This project sits in the same emerging space as recent work on agent reliability, skill evaluation, and agentic QA:

• SkillsBench: Benchmarking Agent Skills: Benchmarks whether curated and self-generated skills actually help agent performance across domains, models, and task types.
• Towards a Science of AI Agent Reliability: Proposes reliability as a separate evaluation axis from raw accuracy, organized around consistency, robustness, predictability, and safety.
• Spec-Driven Development: From Code to Contract in the Age of AI Coding Assistants: Argues that AI-assisted software work should begin from explicit contracts, specs, and acceptance criteria rather than informal intent alone.
• Upskill: Generate and Evaluate Agent Skills: Describes generating skills from agent traces and evaluating them for both task quality and token efficiency.
• How to Evaluate and Test Agent Skills: Walks through a practical skill-eval loop built around trigger tests, rubrics, and judge-based automation.
• Agentic Evaluations Workshop: Surveys broader agent-eval concerns including transparency, environment-based testing, robustness, and governance.

License

CC BY 4.0 - free to use, share, and adapt with attribution.