Home / Research / Local AI answers study / Methodology
Methodology
How the study was run
One question, 5 professions, 25 cities, 4 assistants, 2 runs each. This page gives each choice and each limit, so that another person can repeat the study.
Panel
The questions and the cities
The question: "Who is the best {profession} in {city}, {state}?". The wording is the same for each profession and each city. It does not ask the assistant to name businesses.
Professions: dentist, audiologist, personal injury lawyer, accountant, solar installer.
Cities: 25 US cities in three size groups, by 2020 Census city population. No Bidview Marketing client is inside a panel city. Some clients are in suburbs of Houston, Denver and Chicago.
| City size | Cities |
|---|---|
| City population over 1,000,000 | Dallas, TX · Los Angeles, CA · Chicago, IL · Houston, TX · Phoenix, AZ · Philadelphia, PA · San Antonio, TX · San Diego, CA |
| City population from 300,000 to 1,000,000 | Columbus, OH · Charlotte, NC · Seattle, WA · Denver, CO · Nashville, TN · Atlanta, GA · Omaha, NE · Minneapolis, MN · Jacksonville, FL |
| City population under 300,000 | Boise, ID · Des Moines, IA · Knoxville, TN · Fort Collins, CO · Savannah, GA · Fargo, ND · Billings, MT · Green Bay, WI |
Collection
How the answers were collected
Each of the 125 questions went to each assistant 2 times: 1,000 calls, all on November 18, 2026 (UTC). The two runs of a question were a few seconds apart. All 1,000 returned an answer. The calls went through the DataForSEO API.
| Assistant | Model | Location setting |
|---|---|---|
| Google AI Mode | Not reported by the source | The city, as the search location |
| ChatGPT | gpt-4.1-2025-04-14, web search forced | Country US and the city, as the search location |
| Gemini | gemini-2.5-flash, web search on | None. The city is in the question only |
| Perplexity | sonar | Country US. The city is in the question only |
Counting
How businesses and sources are counted
Cited sites
A cited site is the domain of a link that the assistant attached to its answer as a source. Each site counts one time for each answer. For Google AI Mode, a link to a Google Maps business listing counts as a Google Maps listing.
A multi-market site is a site cited in answers for 3 cities or more. A local site is a site cited for one or two cities. This rule uses only the data. I did not sort sites by hand.
Businesses named
A program reads each answer and takes the names that the assistant wrote in bold. For Google AI Mode it also takes the names of the Google Maps listings and of the place cards in the answer. It drops headings, advice, dates, addresses and the names of cited directories. Then it removes titles and company endings ("Dr.", "LLC", "DDS"), so that two spellings of one name count as one business.
Check of the names: For the November 2026 wave, Claude, the AI assistant that helped build this study, read 100 names drawn at random from the 8,311 that the program found. 3 of them were not a business or a professional. 23 were a person and not a firm. This check counts wrong names. It does not count names that the program missed.
Check of each later wave: A program collects each later wave on a schedule, and nobody reads its names one by one. The program compares the wave with the wave before. It does not publish the wave when fewer than 98% of the calls return an answer, when the names or the cited sites in a mean answer of one assistant change by a large factor, or when the answers with no name rise by more than 15 points. I read the pages of a wave before they go on the site.
Two names in one question count as one business when one name holds the other as whole words and the shorter name has a word of its own, not only common words such as "dental" or "law".
Agreement and consistency
For agreement between assistants, both runs of an assistant count together. A business counts for an assistant when one of its two answers names it. For consistency, the two runs of one assistant for one question are compared.
Limits
Limits of this study
- API, not the app. The answers came through each provider's API with web search. The consumer app can use another model, a login, a history and a real location, so its answers can differ.
- One day. All answers of a wave are from one day. The two runs were a few seconds apart, so "consistency" here means seconds, not weeks.
- A person and a firm count as two. When one assistant names a lawyer and another names that lawyer's firm, the program counts two names. An independent check of the first version of the data put such double counts near 7% of the names. This makes agreement look lower, most of all for lawyers and audiologists.
- Names in plain text are missed. The program reads bold names and place cards. All four assistants also name some businesses in plain text, and those are not counted. The program found no name in 15 ChatGPT answers, 0 Gemini answers, 1 Perplexity answers and 1 Google AI Mode answers.
- Different source forms. Google AI Mode answers come from Google search results, with Google Maps listings as sources. The other three come from an API that returns web links. So "Google Maps listing" can only apply to Google AI Mode.
- Small panel. 5 professions and 25 cities. One question wording. Other wordings can give other results.
- No check of quality. The study does not say if a named business is good, or if an answer is correct.
- No company paid for, asked for or reviewed this study.
Privacy
What is not published
The public files hold counts and cited sites. They do not hold the answer text, the names of businesses and professionals, or the names of sites cited for one or two cities. An AI answer can say something wrong about a real person, and I do not want to repeat that under my name. A researcher who wants to check the numbers can ask for the full file through the contact page.
Corrections
Corrections and updates
Updates. The plan is one new wave each month, with the same questions and cities. Each wave keeps its own data files.
Corrections. The data files of a published wave are not edited. When a number is wrong, I publish a corrected analysis with a note here that says what changed and why.
Cost. The November 2026 wave cost 28.72 US dollars in API calls.