title: AI Natural Language Tests colorFrom: blue colorTo: green sdk: gradio sdk_version: "6.25.0" python_version: "3.10" app_file: app.py pinned: false

Try It Live

ai-natural-language-tests on Hugging Face Spaces — Try the platform directly in your browser without installation.

Table of Contents

Overview

The platform translates natural language requirements into executable E2E tests for:

flowchart LR
  A[AI-Powered E2E Test Generation]
  A --> C[Cypress<br/>.cy.js<br/>Traditional and prompt-powered]
  A --> P[Playwright<br/>.spec.ts<br/>TypeScript async/await]
  A --> W[WebdriverIO<br/>.spec.js<br/>Mocha with Jest-like expect]
  A --> M["Appium ⚠️<br/>.spec.js<br/>Mobile/iOS/Android"]

  style A fill:#e3f2fd,color:#333333,stroke:#666666
  style C fill:#c8e6c9,color:#333333,stroke:#666666
  style P fill:#ffcdd2,color:#333333,stroke:#666666
  style W fill:#ffe0b2,color:#333333,stroke:#666666
  style M fill:#f8bbd0,color:#333333,stroke:#666666

It supports both local engineering workflows and automated pipeline execution. The generator uses contextual data from live HTML analysis and historical pattern matching to produce stable, maintainable test assets.

Quick Start (5 Minutes)

git clone https://github.com/aiqualitylab/ai-natural-language-tests.git
cd ai-natural-language-tests
python -m venv .venv
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
npm ci
npx playwright install chromium

Create .env and set at least one provider key:

OPENAI_API_KEY=your_key

Generate and run one Playwright test:

python qa_automation.py "Test login with valid credentials" --url https://the-internet.herokuapp.com/login --framework playwright --run

Business Value

[!NOTE]

• Reduces manual test authoring effort and onboarding time.
• Standardizes generated test structure across teams.
• Improves reuse through vector-based pattern memory.
• Supports enterprise delivery with CI/CD and Docker workflows.
• Enables faster root-cause diagnosis using AI-assisted failure analysis.

Core Capabilities

[!IMPORTANT] Appium Experimental Status: Appium support is production-ready in code generation but requires:

• External Appium Server: Running separately (e.g., appium --port 4723)
• Mobile Infrastructure: Android emulator/device OR iOS simulator/device
• Platform SDKs: Android SDK, Xcode (iOS), or connected devices
• Environment Setup: ANDROID_HOME, platform-tools, emulator

Test generation works immediately; test execution depends on infrastructure availability.

Architecture

graph TB
    subgraph "User Input"
        A[Natural Language<br/>Requirements]
        B[URL/HTML Data<br/>--url flag]
    C[CLI Requirement Text<br/>one or more prompts]
    end

    subgraph "AI & Workflow Engine"
        D[LangGraph Workflow<br/>5-Step Process]
        E[Multi-Provider LLM<br/>OpenAI / Anthropic / Google]
        F[Vector Store<br/>Pattern Learning<br/>FAISS + SQLite]
    end

    subgraph "Framework Generation"
        G{Cypress Framework}
        H{Playwright Framework}
        W{WebdriverIO Framework}
        I[Cypress Tests<br/>.cy.js files<br/>Traditional & <br/>cy prompt]
        J[Playwright Tests<br/>.spec.ts files<br/>TypeScript]
        X[WebdriverIO Tests<br/>.spec.js files<br/>Mocha + expect]
    end

    subgraph "Execution & Analysis"
        K[Cypress Runner<br/>npx cypress run]
        L[Playwright Runner<br/>npx playwright test]
        M[AI Failure Analyzer<br/>--analyze flag<br/>Multi-Provider LLM]
        P[WebdriverIO Runner<br/>npx wdio run]
    end

    A --> D
    B --> D
    C --> D
    D --> E
    E --> F
    F --> D
    D --> G
    D --> H
    D --> W
    G --> I
    H --> J
    W --> X
    I --> K
    J --> L
    X --> P
    K --> M
    L --> M
    P --> M

    style D fill:#e3f2fd,color:#333333,stroke:#666666
    style E fill:#f3e5f5,color:#333333,stroke:#666666
    style F fill:#fff3e0,color:#333333,stroke:#666666
    style G fill:#c8e6c9,color:#333333,stroke:#666666
    style H fill:#ffcdd2,color:#333333,stroke:#666666
    style W fill:#ffe0b2,color:#333333,stroke:#666666

Workflow

flowchart TD
    A[Start: User Input<br/>Requirements + Framework] --> C[Step 2: Fetch Test Data<br/>Analyze URL/HTML<br/>Extract Selectors<br/>Generate Fixtures]
    C --> D[Step 3: Search Similar Patterns<br/>Query Vector Store<br/>Find Matching Test Patterns<br/>From Past Generations]
    D --> E[Step 4: Generate Tests<br/>Use AI + Patterns<br/>Create Framework-Specific Code<br/>Cypress, Playwright, or WebdriverIO]
    E --> H[HITL Approval<br/>Optional --approve before save]
    H --> F[Step 5: Run Tests<br/>Execute via Framework Runner<br/>Optional --run flag]
    F --> R[Replay Snapshot<br/>--list-html-replays / --replay-html-analysis]
    R --> G[End: Tests Executed<br/>Ready for CI/CD]

    style A fill:#e1f5fe,color:#333333,stroke:#666666
    style C fill:#c8e6c9,color:#333333,stroke:#666666
    style D fill:#ffcdd2,color:#333333,stroke:#666666
    style E fill:#f3e5f5,color:#333333,stroke:#666666
    style F fill:#e8f5e8,color:#333333,stroke:#666666
    style G fill:#f3e5f5,color:#333333,stroke:#666666

Generation follows a deterministic five-step flow:

Technology Stack

Repository Structure

ai-natural-language-tests/
|-- .github/
|-- docs/
|-- cypress/
|   |-- e2e/
|   |   |-- generated/
|   |   `-- prompt-powered/
|   `-- fixtures/
|-- tests/
|   `-- generated/
|-- webdriverio/
|   `-- tests/
|       `-- generated/
|-- web/
|-- webdriverio/
|   `-- tests/
|       |-- generated/
|       |-- appium-tests/
|       `-- prompt-powered/
|-- prompt_specs/
|-- skills/
|-- services/
|-- knowledge/
|-- generated_exports/
|-- rag_failure_analysis/
|-- vector_db/
|-- qa_automation.py
|-- qa_config.py
|-- qa_runtime.py
|-- qa_workflow.py
|-- qa_refinement.py
|-- test_qa_refinement.py
|-- ragas_evaluator.py
|-- ragas_nlp_evaluator.py
|-- cypress.config.js
|-- playwright.config.ts
|-- wdio.conf.js
|-- package.json
|-- package-lock.json
|-- requirements.txt
|-- Dockerfile
|-- docker-compose.yml
|-- agent.yaml
|-- SOUL.md
|-- RULES.md
|-- PROMPT_UPDATE_GUIDE.md
|-- CONTRIBUTING.md
|-- CHANGELOG.md
`-- README.md

Prerequisites

Installation

git clone https://github.com/aiqualitylab/ai-natural-language-tests.git
cd ai-natural-language-tests
python -m venv .venv
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
npm ci
npx playwright install chromium

OPENAI_API_KEY=your_key

$env:OPENAI_API_KEY = "your_key"

Optional: GitAgent (Repo-Specific)

This repository includes a targeted gitagent setup for its QA automation workflow:

• agent.yaml (manifest)
• SOUL.md and RULES.md (behavior and constraints)
• knowledge/ (framework and repo references)

In short: agent.yaml defines the repo agent, SOUL.md and RULES.md define how it should behave, and knowledge/ gives it project-specific framework guidance.

Quick commands:

npm run gitagent:validate
npm run gitagent:info
npm run gitagent:export

git clone https://github.com/aiqualitylab/ai-natural-language-tests.git
cd ai-natural-language-tests
docker compose build

docker compose run --rm test-generator "Test login" --url https://the-internet.herokuapp.com/login

docker compose run --rm test-generator \
  "Test login" --url https://the-internet.herokuapp.com/login --framework playwright --run

GitHub Registry (GHCR)

Pre-built Docker images are published to GitHub Container Registry. No local clone or build required.

Pull and run

docker pull ghcr.io/aiqualitylab/ai-natural-language-tests:latest

docker run --rm \
  -e OPENAI_API_KEY=your_key \
  ghcr.io/aiqualitylab/ai-natural-language-tests:latest \
  "Test login" --url https://the-internet.herokuapp.com/login

Image tags

For publishing and release management, see CONTRIBUTING.md.

Configuration

OPENAI_API_KEY=your_key
ANTHROPIC_API_KEY=your_key
GOOGLE_API_KEY=your_key

# Ollama (default: http://localhost:11434/v1)
OLLAMA_BASE_URL=http://localhost:11434/v1
OLLAMA_MODEL=llama2

# OpenAI-compatible endpoint (vLLM, LM Studio)
LOCAL_OPENAI_BASE_URL=http://localhost:8000/v1
LOCAL_OPENAI_MODEL=gpt-3.5-turbo

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --llm ollama
python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --llm local-openai

ollama run hf.co/aiqualitylab/ai-natural-language-tests

# Use the local fine-tuned model instead of a cloud API
# Note: :latest tag is optional and applied by default
OLLAMA_MODEL=hf.co/aiqualitylab/ai-natural-language-tests
OLLAMA_BASE_URL=http://localhost:11434/v1

python qa_automation.py "Test login with valid credentials" \
  --url https://the-internet.herokuapp.com/login --framework playwright --run

OTEL_PROVIDER=grafana
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-eu-north-0.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64(instance_id:api_token)>

GRAFANA_LOKI_URL=https://logs-prod-eu-north-0.grafana.net
GRAFANA_INSTANCE_ID=<instance_id>
GRAFANA_API_TOKEN=<logs_write_token>

[!TIP] Privacy-first setup:

• Use an LLM provider account/plan that guarantees no training or zero-retention for API data.
• Keep OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, GRAFANA_LOKI_URL, GRAFANA_INSTANCE_ID, and GRAFANA_API_TOKEN unset to avoid telemetry/log shipping.
• Use masked or synthetic test data for sensitive fields.

[!TIP] Need to support a new URL and tune prompts safely? Follow the step-by-step guide in PROMPT_UPDATE_GUIDE.md.

Usage

Quick Reference

Demo & Test URLs

Use these URLs to try the platform immediately:

Quick test command:

python qa_automation.py "Test login with valid credentials" \
  --url https://the-internet.herokuapp.com/login \
  --framework playwright \
  --run

[!TIP] If your global python misses project dependencies, run with the repository virtual environment:

• PowerShell: .\.venv\Scripts\python.exe qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework playwright --run
• Bash: ./.venv/Scripts/python.exe qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework playwright --run

[!NOTE] The current CLI supports URL-driven generation via --url. A direct --data JSON input flag is not implemented in this repository yet.

Natural Language Prompt Examples

[!TIP] Writing effective AI requirements

• Be specific about the action: "Test login" vs "Test login with valid credentials and verify dashboard loads"
• Mention the expected outcome when it matters: "...and verify error message appears"
• Use --url to give the AI real page context — it reads the HTML and picks the right selectors automatically
• Chain multiple requirements in one run: "Test login" "Test logout" --url <url>

Generate Cypress Test

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login

Generate Playwright Test

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework playwright

Generate WebdriverIO Test

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework webdriverio

Generate Appium Test (Experimental)

[!WARNING] Appium test generation works immediately, but execution requires external Appium server + mobile device/emulator running.

# Start Appium server in separate terminal
appium --port 4723 &

# Generate test (works without device)
python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework appium

Appium Setup & Execution (Experimental)

npm install -g appium@latest
appium driver install uiautomator2   # Android
appium driver install xcuitest       # iOS

# Windows PowerShell
$env:ANDROID_HOME = "C:\Users\YourUser\AppData\Local\Android\Sdk"
$env:PATH += ";$env:ANDROID_HOME\platform-tools"

# macOS/Linux
export ANDROID_HOME=~/Library/Android/sdk
export PATH=$PATH:$ANDROID_HOME/platform-tools

# Terminal 1: Start Appium server
appium --port 4723

# Terminal 2: Generate and run test
python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework appium --run

APP_PLATFORM=ios python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework appium --run

Prompt-Powered Cypress Mode

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --use-prompt

Generate and Execute

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework playwright --run

Conversational Test Refinement

Refine generated tests in the UI with natural language instructions — multi-turn refinement supported.

1. Generate tests using the "Generate" button
2. Describe your refinement in the "Refine (describe the change)" textbox
3. Click "Refine" to apply the refinement
4. Repeat for multi-turn refinement

Refinement Examples:

Safety Guarantees:

• File integrity: LLM cannot add, remove, or rename files
• No empty tests: Every file must have runnable code
• No panel corruption: If refinement fails, original code is preserved
• Multi-turn support: Each refinement builds on the previous output

How It Works:

1. Generated code panel displays with // FILE: <path> headers
2. User enters refinement instruction (e.g., "also test the error message")
3. System sends current code + instruction to LLM
4. LLM returns complete revised code (never diffs or fragments)
5. Panel updates and files written to disk (if originals had headers)
6. Repeat for additional refinements

[!TIP] Best Practices for Refinements

• Start with generation, then refine iteratively
• Be specific: "add assertion for button text" vs "improve the test"
• One instruction per refinement (easier for LLM to apply correctly)
• Use the same LLM provider throughout a refinement session for consistency

Failure Analysis

python qa_automation.py --analyze "CypressError: Element not found"
python qa_automation.py --analyze -f error.log

[!NOTE] The AI failure analyzer returns a structured diagnosis:

Field	Description
CATEGORY	Error type: SELECTOR, TIMING, ASSERTION, NETWORK, STATE, NAVIGATION, INTERACTION, CONFIGURATION, ENVIRONMENT, or DYNAMIC_URL
REASON	Root cause explanation in plain English
FIX	Suggested code change or configuration fix

Test Code Review (AI Judge)

Review generated test code quality before execution. The AI judge scores tests on four dimensions and suggests concrete improvements.

Available in Gradio UI:

1. Generate tests using the "Generate" button
2. Review generated code in the "Generated Test Code" panel
3. Click "Review (AI judge)" button
4. Read scores and issues in "Review Scores" panel

Review Dimensions

Scoring & Verdict

• Each dimension scored 0–5
• "needs_work" verdict if ANY dimension scores ≤ 2
• "approve" verdict if all dimensions score > 2
• Issues list: up to 5 concrete problems, most important first
• Refinement suggestion: one-sentence fix for the biggest issue

Example Review Output

{
  "scores": {
    "assertions": 4,
    "selectors": 3,
    "structure": 5,
    "determinism": 2
  },
  "verdict": "needs_work",
  "issues": [
    "Add explicit waits before assertions to ensure determinism",
    "Use data-testid attributes instead of nth-child selectors",
    "Add error case assertion for failed login"
  ],
  "refinement_instruction": "Replace hardcoded timeouts with proper element waits and data-testid selectors."
}

Works with all frameworks

Code review applies uniformly to Cypress, Playwright, WebdriverIO, and Appium tests. The AI understands framework syntax from the framework context and provides actionable feedback.

Unit Tests

The repository includes a minimal, production-ready unit test suite that validates core, dependency-free logic with zero external API calls or secrets required.

Test Coverage

Running Tests Locally

# Install dependencies (includes pytest)
pip install -r requirements.txt

# Run all tests
pytest tests/unit_tests.py -v

# Run a specific test class
pytest tests/unit_tests.py::TestFrameworkConfig -v

# Run with short traceback
pytest tests/unit_tests.py -v --tb=short

Pattern Inventory

python qa_automation.py --list-patterns

AI Quality Evaluation (Ragas)

The project ships two evaluation scripts that measure whether the AI-generated tests are actually grounded in the page under test. They run automatically in CI but can also be run locally at any time.

Why these scripts exist

Generating test code from natural language is only useful if the generated tests match what the page actually contains. Without evaluation, you have no signal on whether GPT-4o-mini produced a test that talks about the right form fields, the right URL, or the right expected outcomes. These two scripts give that signal using Ragas, a framework designed to evaluate LLM outputs.

ragas_nlp_evaluator.py — No API key needed

This script compares the AI's generated text against reference answers using three classical NLP metrics that run fully offline.

The overall score for each sample is the average of all.

Run locally:

pip install ragas==0.4.3 sacrebleu rapidfuzz rouge-score
python ragas_nlp_evaluator.py

By default it reads test_dataset.json. Each entry in that file needs:

{
  "name": "login valid",
  "response": "The generated text from the AI",
  "reference": "The expected correct answer"
}

Pass a custom dataset or threshold:

python ragas_nlp_evaluator.py --dataset my_dataset.json --threshold 0.60

The script exits with code 1 if any sample scores below the threshold, which lets CI block on low-quality output. It also prints min, max, and overall average at the end.

ragas_evaluator.py — Requires OPENAI_API_KEY

This script evaluates the generated test file against the live page HTML using four LLM-based Ragas metrics.

It fetches the page, asks GPT-4o-mini to answer the requirement using page HTML, and scores how well that answer holds up.

Run locally:

python ragas_evaluator.py "Test user login with valid credentials" \
  --test cypress/e2e/generated/login.cy.js \
  --url https://the-internet.herokuapp.com/login

How they fit into CI

CI runs ragas_nlp_evaluator.py first (no API key, fast) as a quality gate. If it passes, the three framework jobs run in parallel — each generates tests, then calls ragas_evaluator.py against every generated file before running the tests themselves.

nlp-baseline job
  1. Install nlp dependencies
  2. Run ragas dataset baseline on test_dataset.json

if baseline passes
  test matrix jobs run in parallel
  1. Install dependencies
  2. Generate tests
  3. Run ragas generated test evaluation per file
  4. Run tests
  5. AI failure analysis if tests fail

CI/CD Integration

flowchart TD
  A[Code changes pushed to repo] --> B[Pipeline starts]
  B --> C[Install dependencies]
  C --> D[Run ragas dataset baseline]
  D --> E[Generate tests]
  E --> F[Run ragas generated test evaluation]
  F --> G[Run tests]
  G --> H[Tests pass]
  G --> I[Tests fail]
  H --> J[Deploy application]
  I --> K[Analyze failures]
  K --> L[Regenerate tests]
  L --> G
  K --> M[Notify developers]

Recommended pipeline stages:

Security and Compliance Guidance

[!IMPORTANT]

• Store secrets only in secure secret managers (never commit .env).
• Use scoped API tokens with least-privilege access.
• Rotate provider keys and Grafana tokens on a fixed cadence.
• Keep generated tests and reports free of sensitive production data.
• Apply repository protection rules and mandatory CI checks.

Troubleshooting

[!WARNING] Traces Not Visible in Grafana Tempo

• Verify OTLP endpoint region and datasource selection.
• Verify Authorization=Basic <base64(instance_id:api_token)> format.
• Query with:

{resource.service.name="ai-natural-language-tests"}

[!NOTE] Loki Authentication Errors

• Ensure token has logs:write scope.
• Confirm instance ID and logs endpoint match the same Grafana stack.

[!NOTE] Loki 5xx Transport Errors (for example, 502)

• Test generation continues; logging transport failures are treated as non-blocking.
• The runtime suppresses traceback spam and emits a throttled warning instead.
• Investigate Grafana stack health/network path if these warnings persist.

[!TIP] Docker Observability Validation

• Confirm .env includes OTLP and Loki keys before docker compose run.
• Use docker compose config to verify environment interpolation.
• In Grafana Explore, query Tempo with service.name="ai-natural-language-tests".
• In Grafana Loki, query labels: {service_name="ai-natural-language-tests"}.

[!TIP] Switching to Headed Mode for Debugging

Tests run headless by default. To debug interactively, switch your framework config:

Cypress:

• Edit cypress.config.js and add headed: true after browser: 'chrome'
• Or run: npx cypress run --headed --spec 'cypress/e2e/generated/*.cy.js'

Playwright:

• Edit playwright.config.ts and change headless: true → headless: false
• Or run: npx playwright test --headed tests/generated/

WebdriverIO:

• Edit wdio.conf.js and comment out '--headless=new' from the args array

Docker Headed Mode (with X11 forwarding):

docker build --target debug -t ai-tests:debug .
docker run -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix ai-tests:debug

• Optional: mainly for Linux visual debugging.

• Retry with generated single-spec command from logs.

Documentation Map

Versioning and Release Policy

Support and Security Reporting

Compliance and Data Handling

For implementation details and contribution controls, see CONTRIBUTING.md.

[!NOTE] This project follows EU AI Act readiness practices. See MODEL_CARD.md and INCIDENT_RESPONSE.md.

Operational Expectations

These are practical runbook-style expectations for delivery teams. They are operational targets, not contractual SLAs.

Support Matrix

The matrix below reflects currently configured and documented project baselines.

Changelog

Release notes are maintained in CHANGELOG.md, following the Keep a Changelog format.

Production-focused AI-assisted E2E test generation for modern QA teams.

© 2026 AI Quality Lab / Sreekanth Harigovindan
tests.aiqualitylab.org