AI Recommendation Consistency Dataset

How stable are the business recommendations you get from web-search-enabled LLMs when you ask the same question more than once?

This repository releases the full response data and analysis code from an exploratory study of run-to-run consistency in ChatGPT and Google Gemini. We asked ten open-ended "best X for a small business" questions, three times each, to each assistant on a single day (60 responses total), extracted the set of businesses each answer recommended, and measured how much that set changed between identical repeated queries.

The short version: about one recommended business in three changed when the same question was asked again minutes later. But the instability is structured, not random. The top one to three names stay locked in; the lower-ranked slots rotate.

This is a small, deliberately-scoped study. The numbers are directional, not definitive, and we are upfront about the limits below. The reason we are releasing every response is so anyone can check the work.

---

Headline findings

The structural finding is the interesting part. A single "AI is 69% consistent" number hides the shape. The leading recommendations for a given question recur across every run (a stable core), while the fourth, fifth, and sixth positions shuffle each time (a volatile tail). For a smaller or newer business, the positions you could realistically reach are exactly the least stable ones.

---

What's in this repo

Reproduce the headline stats:

node analyze.mjs data/study_final_2026-09-13.json

---

Methodology

Questions

Ten open-ended "best/top X for a small business" questions, one per commercial category:

We used open questions on purpose. Head-to-head comparison prompts ("X vs Y") were deliberately excluded: they can only return the two names you put in the prompt, so they trivially maximize apparent consistency and would inflate the topline. Open "best X" questions are also exactly the case a business wants to be recommended in.

Assistants and procedure

• ChatGPT via the OpenAI Responses API with the web_search tool enabled.
• Google Gemini via the Generative Language API with Google Search grounding enabled.

All queries specified a United States locale and English responses. Each question was issued three independent times to each assistant, yielding 60 responses. All responses were collected on 2026-09-13 within a single session to hold model versions roughly constant. On this run, all 60 responses were grounded in a live web search (verified per response, not assumed).

Extraction

From each response we extracted the set of specific businesses recommended, using an automated language-model extraction pass instructed to include only named businesses presented as options, and to exclude media outlets, generic categories, and services named only as context.

We validated the extraction against the source text: all 290 extracted business names appear verbatim in the response they were drawn from (zero fabricated names). You can check this yourself, every answer's full text is in the dataset.

Consistency measure

For each question and assistant, consistency is the mean pairwise Jaccard similarity of the recommended-business sets across the three repetitions:

J(A, B) = |A ∩ B| / |A ∪ B|

We count only repetitions that produced a shortlist of two or more businesses, so an empty or single-item answer can never masquerade as a "different recommendation." Brand names are normalized (lowercased, non-alphanumerics stripped) before set comparison.

We separately count, per question-assistant, how many distinct businesses appeared in all repetitions (the stable core) versus exactly one (the volatile tail).

---

Limitations (read these before you cite)

This is a small exploratory study, and treating it as more than that would be misusing it.

• Ten questions, three repetitions, two assistants, one day. Enough to surface a pattern, not to estimate precise category-level or assistant-level rates. The reported means carry no formal confidence interval, read them as directional.
• Three repetitions quantize the per-question values coarsely (a per-cell value can only land on a few discrete points). One cell (Gemini, e-commerce) rests on two repetitions rather than three, because the third named a single business and was excluded by the two-or-more rule.
• Automated extraction can under- or over-count borderline mentions (for example treating "Wave" and "Wave Accounting" as distinct). This biases toward understating consistency, not inflating it.
• The "most named" brand tally excludes any brand named in a prompt, so a model is never credited for repeating a name we handed it.
• Findings reflect two specific assistants on one day. Assistant behavior evolves; a rerun months later will differ.
• This study was conducted by a commercial entity (Pressfront) with an interest in the subject. That is exactly why the full method and every response are released here: so you don't have to take our word for any of it.

If you rerun it and get different numbers, that's a finding, not a contradiction. Open an issue or a PR.

---

How to reproduce or extend

The ten questions, the three-repetition procedure, and the Jaccard-overlap measure above are sufficient to reproduce the study end to end. analyze.mjs regenerates every published number from the raw JSON. If you have OpenAI and Gemini API keys and want to run a fresh collection (more questions, more repetitions, more engines, a later date), the method is fully specified in this README, and we'd genuinely like to see what you find.

Useful extensions we haven't done: more repetitions per question (to get real confidence intervals), more engines (Perplexity, Claude, Copilot), longitudinal reruns of the same questions over weeks, and per-category sample sizes large enough to compare categories.

---

Citing this dataset

If you use this data, please cite it (see CITATION.cff). The full writeup is in the preprint bundled here: ai-recommendation-consistency-preprint.pdf, and the DOI-registered version is on Zenodo:

Pressfront Research (2026). Run-to-Run Consistency of Business Recommendations from Web-Search-Enabled Large Language Models: An Exploratory Study. Zenodo. https://doi.org/10.5281/zenodo.22738861

DOI: 10.5281/zenodo.22738861 (this version) · 10.5281/zenodo.22738860 (all versions)

For the companion writeup and further analysis: pressfront.co/research

License

Data and code released under CC BY 4.0 — use it freely, including commercially, just credit the source.