demo-e2e

End-to-end smoke tests that drive a live AI booking assistant in a real headless browser, the way a customer would: open the page, type a message, send it, and read what comes back.

Target: https://demo.consentleads.uk

Why this exists

A health check can report green while the page is dead. The process is up, the health endpoint returns 200, the monitoring is quiet, and the chat surface underneath still answers nothing. Nothing in that stack sends a message and reads the reply, so nothing in it notices.

Opening the page and typing into it does notice, and that is what this automates.

The browser path is the one a customer actually takes, so it is the one worth asserting on. The API underneath is asserted separately, because a 200 from the API and a working page are different claims.

What it checks

test_page_shell_renders
    The chat surface exists and the business identity has loaded.
    Catches the case where the HTML serves but /config never resolves,
    leaving a chat box attached to no business.

test_chat_round_trip_returns_a_real_answer
    The whole user path. Types a question, sends it, waits for the
    assistant bubble, and asserts the reply is present, long enough to
    be a real answer, and not the "(error: ...)" bubble the page shows
    when a request fails. A 200 with that text on screen is still a
    broken demo.

test_no_console_errors_during_a_conversation
    A reply can arrive while the page throws underneath it. A CSP
    nonce regression once broke the chat in the browser while every
    server-side check stayed green.

test_chat_api_responds
    The fast path. Calls /chat from the page's own origin, so it
    exercises the deployed route rather than a local guess at the
    contract.

Running it

pip install -r requirements.txt
python -m playwright install --with-deps chromium
python -m pytest

Configuration is by environment variable, so the same suite runs against any deployment:

DEMO_BASE_URL          default https://demo.consentleads.uk
DEMO_VERTICAL          default dental
DEMO_REPLY_TIMEOUT_MS  default 30000

A race in the suite itself, found and fixed

The first version waited for #input to appear and then read the header. It passed. Run again four times, it failed four times:

run 1: name='Bridge Dental Hackney' sector='dental'
run 2: name='Loading...'            sector=''
run 3: name='Loading...'            sector=''
run 4: name='Loading...'            sector=''

#input ships in the initial HTML, so waiting for the element returned before /config had resolved. The first run only passed on timing. The fix is to wait on the STATE rather than the element: wait_for_config polls until the header is populated and no longer reads "Loading...".

This is worth writing down because a suite that passes once and fails four times is worse than one that fails every time. It teaches the team to re-run until green.

The tests were verified to fail

A test that cannot fail is not a test. These were run against a target with no chat surface to confirm the assertions bite:

DEMO_BASE_URL="https://example.com" python -m pytest -x

FAILED tests/test_demo_smoke.py::test_page_shell_renders
TimeoutError: waiting for locator("#input") to be visible

Against the live target the same suite passes in about 12 seconds.

CI

.github/workflows/e2e.yml runs the suite on push, on pull request, and on a weekday morning schedule, so a silent outage is found before anything goes out that points at the demo.

Scope and limits

• These are smoke tests. They prove the path works, not that the assistant gives good answers. Answer quality is a separate problem and is not claimed here.
• The suite reads a public URL and needs no credentials or secrets.
• It asserts on one vertical at a time. The site serves five.

CI caught what the developer machine could not

The first CI run failed on all three browser tests while the same suite passed locally:

EvalError: Refused to evaluate a string as JavaScript because
'unsafe-eval' is not an allowed source of script in the following
Content Security Policy directive: "script-src 'self' 'nonce-...'"

page.wait_for_function evaluates its predicate through eval, and the target ships a strict Content Security Policy with no unsafe-eval. Local was running playwright 1.60, which has a CSP-safe path. CI was running the pinned 1.49.1, which does not.

Two fixes, because one alone would have been luck:

1. wait_for_config now polls text_content from Python. That touches no eval and works on every version, CSP or not.
2. The pin was moved to the version actually tested against.

The lesson is the point of the pin: a green run on one machine is not a green build. The CSP that broke this suite is the same CSP the site is tested for, which is a fair reminder that a test harness is production code and gets to be wrong like any other.