Do Not Force Search in ChatGPT Prompt Tracking

Prompt Tracking Data Often Contains More Citations Than Expected

We recently reviewed several prompt tracking datasets in which an unexpectedly high percentage of ChatGPT responses have citations.

This includes datasets requested from three different companies for research purposes, prompt banks for clients from two prominent prompt tracking tools, and responses from an AI scraping company, all of which had citations in nearly every response.

In our recent internal experiments with logged-out scraping for informational entity-comparison prompts, we find that 58% of responses show evidence of using search (a fan-out query or search results). Separately, a more general Similarweb study found that across all prompt types, 93% of U.S. ChatGPT responses did not have citations.

Why are citations so frequent in some response datasets and not in others?

ChatGPT Sometimes Responds Using Parametric Knowledge, and Sometimes Using Web Search

Often, a model responds to a prompt using only its parametric knowledge, essentially what it learned during pre-training on a large subset of the internet. This parametric knowledge may be out-of-date, or for more niche topics, the model's recollection might be fuzzy.

Therefore, ChatGPT and other AI systems also have access to a web search tool to retrieve additional online references that are relevant to the prompt. When using this tool, the model selects one or more searches, also called fan-out queries. After receiving the search results, the model decides which of the returned results to “open” (read), and the content of those pages is added to the context the model uses to answer. This is also called “grounding” the model's responses. Prior work suggests that grounding can have a large influence on the response and override parametric knowledge.

By default, the model decides when it needs to use the web search tool. Typically ChatGPT will invoke the web search tool when it infers that fresh information is required to answer the prompt.

Users Can Force Web Search, but Most Users Likely Use the Default

A user can also choose to force the use of the web search tool. In the ChatGPT UI, this requires clicking the plus button and selecting “Web search” from the tools menu. In this paper we call this “forcing search.”

We are not aware of public data reporting how often users manually select web search. However, doing so requires opening the tools menu and selecting an additional option, while the default experience requires only submitting the prompt. Extensive research on default effects shows that users disproportionately follow the default path, particularly when an alternative requires additional effort. Given the wide user base, many ChatGPT users likely do not understand how enabling web search would differ from allowing the model to decide. Users who do not manually select Web search may still receive search-grounded responses when ChatGPT determines that search is useful. Therefore, it is reasonable to expect manually selecting search to be less common than simply using the default.

Without Forcing, 58% of ChatGPT Responses Show Evidence of Search

For informational entity-comparison prompts, we find that logged-out ChatGPT currently (August 2026) exposes evidence of using the web search tool for 58% of responses when not forced. We cannot directly observe ChatGPT's internal tool invocation, so we classify a response as using search when there is at least one fan-out query or search result. We do not use the presence of links alone because, as of May 2026, ChatGPT can insert inline links without exposing either of these search signals, as reported by Similarweb and others.

Some Prompt-Tracking Pipelines Appear to Force Web Search

The reason some tools and datasets have citations for every response appears to be that they are forcing the use of web search when collecting responses. For example, in one dataset we reviewed covering the same 100 prompts used in our experiment, each response had at least one search query. In another test, six prompts that showed no evidence of search across 600 default responses were classified as using web search in all 18 responses collected by a popular prompt-tracking tool.

The Motivation for Forcing Search Makes Sense

Cited references form the core of many AEO/GEO strategies, like publishing new content for the model to cite, or doing outreach to get mentioned on offsite pages that are frequently cited. By forcing web search, we always get actionable citation data. Users of these tools likely prefer to always receive citations.

But Forced Search Does Not Reveal Hidden Sources

A potential misconception is that forcing search is helpful because it uncovers influential sources that would usually be hidden. This is not correct.

When ChatGPT responds without search, it relies primarily on information encoded in its model parameters, learned from many examples during training. There is generally no small set of identifiable webpages influencing a particular response. Forcing search retrieves a relatively small set of pages and adds them to the model's context. It changes the information available to the model rather than exposing the provenance of its parametric answer.

A cited page could be included in a future training set, but that is a less direct and less controllable way to influence the model. If included, a cited page would be part of a massive corpus, and there is no reason to assume it would have special weight. Whether a page is included in the training set, how it would affect the model's parametric knowledge, and when that model would be publicly released are all unknown.

The resulting citations answer the question, “Which sources influence the response when we require search?” They do not necessarily answer, “Which sources influence the response users will receive?”

Risk 1: Chasing Citations That Have No Influence

When the model does not use the search tool, the citations obtained when forcing search may have no influence on the unforced response. The risk is that we chase citations that do not actually affect what users see when using ChatGPT.

For example, we found that the prompt “What is the best texting service for business?” triggered web search (without forcing) in 31% of responses. So if we force search and focus our efforts on the resulting citations, we are optimizing for 31% of responses.

Risk 2: Getting Visibility Wrong

Because the presence or absence of grounding changes the responses, brand visibility values (the percentage of responses in which a brand appears) may also vary.

The decision to invoke the search tool is also subject to the same randomness inherent in generation by LLMs. Some prompts will usually trigger search whether or not it is forced. Other prompts will usually not trigger search unless forced. And some prompts will trigger search for some responses but not others.

We use a set of 100 prompts and collect 100 responses for each in two settings: using forced web search and letting ChatGPT decide whether to search (unforced). We compute visibility values (using the same methodology as in our previous paper Demystifying Randomness in AI) for both settings and compare the mean absolute differences. We include in this calculation any entity (e.g., product, brand, company) that has a visibility of at least 10% in at least one of the two settings.

We display below the mean absolute differences bucketed by how often the responses use search when unforced. We can see that the differences are largest for prompts that never use search, and then they decrease for prompts where search is typically used even when not forced.

Note that no prompts in our dataset always use search when unforced. The difference in the 75-99% bucket does not go to zero due to a combination of real differences in whether search is used and sampling noise. We estimate that sampling alone would yield absolute differences of 3.4 to 4.1 points across the buckets.

Next we show a specific example with the prompt “What are the most founder-friendly investment firms?” For many firms, the visibilities are quite different when search is forced.

Therefore, when we force search, we may get visibility values that do not reflect what users see in responses.

General Principle: Prompt Tracking Should Simulate User Behavior

While prompt tracking is essentially a simulation, we want the simulation to match real user behavior. That is, we want to select prompts users are likely to type, simulate running them as a user would, and then optimize so that a particular brand or product is mentioned or cited.

As noted above, because manually forcing search is likely less common than simply using the default, doing so is not a good simulation of real user behavior.

Some users do force web search, and we may want to optimize for them too. So we could track both forced and unforced web search. However, we expect unforced to be by far the more common case.

Recommendation: Do Not Force Search and Report the Prompt Grounding Rate

Due to these risks, we recommend not forcing search and instead letting the model decide when to search. Additionally, we recommend reporting the prompt grounding rate: the percentage of runs in which the prompt naturally triggers search. Many tools seem to aggregate citations across responses, which can make those citations appear representative of the prompt generally even when they occur in only a small subset of responses.

Recommendation: Do Not Rely on Citation-Based Strategies Unless the Grounding Rate Is High

When a prompt does not consistently trigger search (the prompt grounding rate is not high), recognize that strategies focusing on citations are unlikely to be effective.

Limitations and Future Work

The experiments in this paper use only logged-out scraped data, and logged-in ChatGPT behavior could differ. As part of ongoing research, we collected responses to the same prompts from real logged-in users in April 2026. Since this data was collected before ChatGPT started including citations without search, we use citations to determine whether search was used. Citations appeared in 53.4% of 10,872 responses, providing no indication that logged-in ChatGPT uses search substantially more often than the logged-out ChatGPT studied here.

In future work, we plan to conduct a broader study of citation behavior across models, access modes, and prompt types.