1. Models generate token by token; sampling can make paths diverge
Thinking of a chat model as an “answer box that can look things up” easily leads to error. Even when it answers a familiar question, it does not first choose an entire correct passage and then retrieve it unchanged. During generation, the system uses the input already supplied and the text already generated to form a set of candidates for the next token and their relative likelihoods. Once one has been selected, that newly selected token becomes part of the conditions for the next step. A token is a unit in the generation process; readers need not equate it mechanically with a Chinese character, an English word or a complete judgement. What matters is that an answer is a sequence built up from local choices.
Public Transformers documentation makes a clear distinction between two common strategies: greedy search selects the most likely next token at each step, while multinomial sampling selects according to the probability distribution over the vocabulary, so every candidate with non-zero probability can be drawn. The documentation also says that sampling can reduce repetition and produce more creative, diverse output. It describes a generation mechanism in a public implementation; it does not tell us whether a particular chat product is currently sampling, which parameters it uses, or whether its answer is correct. [1]
This helps to explain a common pattern: two answers differ by only a few words at the start, then drift ever further apart. One might begin, “First check which jurisdiction applies”; another, “First confirm the publication date.” Each opening leads the subsequent generation towards different examples, paragraph orders and qualifications. The widening difference does not automatically mean that the model has changed its mind about some hidden factual question; it may simply be developing the response from a different writing entry point. Likewise, one answer being shorter and another more detailed may arise from different early organisational choices, rather than meaning that the longer answer necessarily received more factual checking.
“Probability” does not mean unstructured invention. It describes how the next unit is selected under the current textual conditions; that selection and whether some fact in the outside world is true are separate matters. A model can generate a false assertion very consistently, and it can express the same externally verified fact in different words. So, when assessing two outputs, do not begin with their similarity alone. Differences can be divided roughly into three layers: wording, word order, analogies and interchangeable examples may differ; the premise, time range or intended group of the question may differ; or verifiable facts, sources, numbers or recommended actions may differ. The first layer will not usually justify calling the model’s response a reversal. The latter two need further checking.
It is also important not to mistake greedy decoding for a synonym for a “fixed answer”. Even if a system favours the highest-probability candidate at a particular step, readers cannot infer that the inputs, model version and service conditions were the same in two requests; those variables are considered below. Conversely, knowing that an output may involve sampling does not by itself show that it is untrustworthy. Generation paths can diverge while factual judgements remain the same. The question is not whether two passages match word for word, but what their divergence changed and whether that change affects the reader’s action.
2. Temperature and truncation controls shape answer diversity
People often explain every different answer with the phrase “the temperature is higher, so it is more random”. That captures part of the phenomenon, but it merges several distinct controls into one mysterious dial. In the public Transformers configuration, temperature adjusts the probability of the next token; top-k retains a fixed number of the highest-probability candidates; top-p retains the smallest high-probability candidate set whose cumulative probability reaches a threshold; and a separate option distinguishes whether sampling occurs. All of these act on the candidate distribution or candidate set during generation, but none is a dial for factual correctness. [2]
Temperature can be understood as one kind of adjustment to how concentrated the distribution is. Within the same implementation, a more concentrated distribution will generally produce choices that fall repeatedly among a smaller number of candidates, while a more diffuse distribution gives more candidates an opportunity to appear. Top-k and top-p instead address which candidates remain on the table. This distinction does not require ordinary users to memorise the parameters, nor does it support applying one “best temperature” to every product. Many chat interfaces do not show these controls at all; even where they do, names, defaults and actual behaviour may differ between models and products. The cautious conclusion is only that decoding settings may alter textual diversity; they cannot on their own explain every disagreement.
That is why low diversity should not be presented as a switch for truth. In open-ended writing, headline rewrites, interview outlines and idea-generation tasks, the aim is not to identify one uniquely correct sentence from several candidates, but to obtain expressions or ideas that can be compared. Research on nucleus sampling discusses how maximisation-based decoding can produce bland, incoherent or repetitive open-ended long text, and proposes sampling from a dynamic high-probability “nucleus” to balance quality and diversity in its setting. The finding shows that determinism is not the sole objective of generation quality; it does not say that high-risk factual questions should seek more randomness. [3]
In some controlled reasoning tasks, multiple paths can even form part of the method. The self-consistency approach is not a matter of asking a question several times and choosing the passage one likes. It samples multiple reasoning paths and then aggregates answers under an explicit rule; the paper reports improvements over its baseline on several arithmetic and common-sense benchmarks. Its limited implication is that diversity may be useful for specific tasks with a specific aggregation method. It does not support treating five answers from an ordinary chat interface as five independent experts, nor treating a majority answer as a factual source. [4]
For creative or divergent tasks, then, the better approach is to retain several candidates and have a person select among them using the audience, factual constraints, tone and goal as criteria; the differences themselves are material. For tax rules, drug use, contractual obligations or safety operations, the better questions are not “How can I turn the temperature down?” but “What primary material does the answer cite, when and where does it apply, and can I check it independently?” More conservative settings may make the language more concentrated, but they cannot perform source review for the reader or eliminate differences caused by changes in input, context and product.
3. Conversation state, instruction hierarchy and tool outputs change the actual input
The same visible sentence does not necessarily amount to the same actual request. The most obvious variable is conversation history. In a continuing conversation, the model may need to process previously discussed task goals, people, jurisdictions, formats, corrections and generated material at the same time. A user may think they are merely asking again, “Is this proposal feasible?”, while the conditions received by the system also include earlier descriptions of budget, deadline or risk preference. A new conversation may lack those conditions. Which is better depends on the purpose: continuing work needs its background; diagnosing why the same question produces different answers requires actively reducing differences in that background.
Public documentation on conversation state helps to explain this. OpenAI’s documentation says that conversations can persist state, and that conversation items can include messages, tool calls, tool outputs and other data; when a response continues from an earlier response, the later response can share context. It also notes that the context window covers inputs, outputs and, for some models, reasoning tokens. What this establishes is how a public API can pass conditions, not that any consumer chat interface necessarily stores the same things or truncates them in the same place. [5]
Inputs have a hierarchy as well as a history. OpenAI’s Conversations API reference lists message roles including user, assistant, system and developer; it says that developer or system instructions take precedence over user instructions, and includes tool-output structures such as function_call_output. That fact is enough to show that the one sentence a user sees may be only part of a complete input structure. It does not allow us to infer a particular product’s undisclosed system instructions, still less to attribute every surprising answer to a backend that secretly changed the prompt. In the absence of public evidence, the strongest conditional statement is this: if a product adds higher-priority rules, tool definitions or other context, copying only the visible sentence still leaves variables uncontrolled. [6]
Attachments and tools make the distinction more concrete. A contract, spreadsheet or meeting minutes uploaded by a user may cause an answer to be organised around its text. When browsing, retrieval or other tools are enabled, the web pages, file excerpts or data returned by the tools may also differ at different times. In that situation, two answers citing different links need not be the result of language sampling alone: the input material or live retrieval results may already have changed. Conversely, identical tool results still do not prove that the conclusion is correct; they merely hold one potential variable constant for the moment.
This gives repeat testing a plain but often overlooked principle: first say what you want to compare. To compare whether the same visible question is stable under conditions that are as similar as possible, use independent new conversations, remove unrecorded attachments, fix the language, requested output format and visible feature switches, and save the time. To investigate whether history affects the answer, change only history and record all other conditions. If the model, tools, attachments, conversation and wording of the question all change at once, comparing the two answers yields only results from two different experimental conditions; it cannot reliably attribute the difference to any one of them.
For ordinary users, this does not mean turning every chat into a laboratory exercise. It means changing one habit: when a second answer adds an exception, changes the jurisdiction or cites something different, first ask, “Are the materials brought into this answer the same as last time?”, rather than immediately choosing the more confident or longer version. For work that needs an audit trail, preserving the prompt text, attachment version, whether tools were used and the position in the conversation often explains later differences better than repeatedly polishing the question.
4. Model versions and service configuration can change how the same prompt behaves
Even if the wording of the question, the conversation and visible settings are held as constant as possible, time remains a variable. An unchanged product name does not mean that the underlying model snapshot, post-training behavioural boundaries or service configuration will remain unchanged. OpenAI’s API reference explicitly notes that prompt behaviour can change between model snapshots and that outputs themselves are variable; for more consistent prompt behaviour and outputs, it recommends using a fixed model version and conducting evaluations. This is a public compatibility statement about that API. It cannot be extrapolated into a claim that every service names or updates models in the same way. [7]
Some technical users may try to reproduce a request with controls such as a seed. An archived OpenAI example says that, within its then-current Chat Completions and Completions scope, fixing the seed, request parameters and system_fingerprint can make outputs mostly consistent, while offering no guarantee of determinism. There, system_fingerprint identifies changes in model weights, infrastructure and server configuration that may affect determinism. Two qualifications cannot be omitted: the page itself is archived, so its current support scope may differ; and reproducing an output and establishing that the output was correct remain wholly different matters. [8]
Model lifecycle is not merely theoretical. Anthropic’s platform documentation says that old models are regularly retired as new models are released, and recommends thorough testing before applications that depend on them migrate to replacements. This example proves only that version replacement and migration testing are public variables in real product operations; it does not show which internal path handled a particular chat request. Its value is as a reminder: when answers are compared across days, weeks or an update, first treat that as a signal that the reproducible experimental conditions may have changed, rather than trying to infer an undisclosed provider decision from the finished answer. [9]
The appropriate record depends on the use case. API users with the necessary access and visible fields can record the model snapshot, parameters, request time, request ID and any available system identifier; those records help with review and troubleshooting. Ordinary chat users generally cannot obtain all of the technical details, but can still note the model name shown in the interface, date and time zone, whether browsing or file features were enabled, visible version notices and the full answer. Records need not pretend to be precise about internal details that cannot be observed. Their purpose is to help later readers distinguish “the model phrased it differently” from “the test conditions changed”, and to identify questions that cannot be resolved from the user interface alone.
5. Reliable use in consequential decisions depends on stability testing, source verification and version records
The most valuable aspect of repeating a prompt is not holding a vote among answers, but revealing the conditions under which an answer changes. For writing tasks that do not affect real-world action, that finding can help select a more suitable version. For questions that can affect health, law, finance, safety, employment or other important decisions, it should trigger stricter verification. NIST’s generative-AI risk-management material recommends assessing output accuracy, quality, reliability and truthfulness against known-fact benchmarks and through multiple evaluation methods, and deploying and documenting fact-checking techniques, especially when information comes from multiple or unknown sources. This is a voluntary framework for organisations and AI actors; it does not make medical, legal or investment decisions for an individual. It does support a simple principle: do not compare only the style of answers or the majority view; compare verifiable facts and primary material too. [10]
For an initial diagnosis, readers can use the article’s own “5×2 repeat-prompt record method”. It is not a research finding and does not promise to measure real accuracy. It simply turns the vague instruction to “ask again” into a comparison that can be revisited. First choose a question that affects an actual judgement but whose authoritative primary material can later be found. Put the jurisdiction, affected group, cut-off date, desired output format and a request to “list the original sources” into the same prompt. The more specific the question, the less room there is for the model to supply different unstated premises on its own; a specific prompt still cannot guarantee that the material it gives is true or complete.
For the first group, use five mutually independent new conversations. Copy the same prompt word for word each time. If the interface allows the model, language, answer length, browsing or other tools to be selected, fix and record those visible conditions, and also record whether there are no attachments. Treat that prompt, model selection, language, attachment state, tool switches and output requirements as the common baseline. Before sending the prompt in each of the five conversations, restore that baseline; do not inherit the preceding answer or make an ad hoc adjustment. Save the complete answer, date and time, and any accessible source links in it. The purpose of independent new conversations is not to pretend that every hidden variable has been eliminated, but to avoid making one answer the background for the next. Five is not a magic sample size; it merely helps prevent one chance fluent expression from being mistaken for a stable pattern.
For the second group, again make five mutually independent repetitions, and restore the common baseline documented in the first group before each one. To test just one visible variable, choose “fixed background material”: create five new blank conversations, copy the same prompt, model selection, language, tool switches and output requirements into each, then attach to each conversation one file of the same version and exactly the same content. Relative to the first group, the only deliberate change is that file; within the second group, the file and every other condition must remain identical across the five repetitions. Do not make five successive additions in one conversation with an accumulating history, because the previous answer will become context for the next and the change will no longer be limited to one variable. Within the group, also confirm that the file content, name and version marker remain unchanged. If an upload fails, the tool state changes or the interface automatically introduces new information, discard that record and start again from the common baseline; do not append it to a conversation that has already produced an answer.
When comparing, do not place five passages side by side and decide by feel. Mark four things in every answer: its final conclusion; the key facts, figures and dates on which it relies; the original sources it cites; and the action it recommends, conditions it preserves or assistance it suggests seeking. Where materials are available, open the underlying laws, regulator guidance, product documentation, original research or institutional notices directly, rather than comparing only the model’s summaries and link titles. API users can also fix the model snapshot, parameters and seed, and save the system_fingerprint, if the service offers those controls. Those measures can improve the consistency of the conditions for review, but they cannot substitute for an assessment of correctness. [8]
This record makes the distinction between “normal variation” and variation that needs investigation clearer. Normal candidates include different wording, examples, orderings and interchangeable writing proposals, while verifiable facts, scope of applicability, sources and action conclusions remain the same. Differences that need investigation include a premise being quietly rewritten, a different jurisdiction or time range, conflicting key figures, a citation shifting from primary material to a secondary account, or one version omitting a qualification present in another. First identify whether the change is more likely connected to sampling, conversation, tool results or version, then return to the primary material for verification. Do not let the model use a third answer to decide a conflict between the first two.
The warning is clearer still when, in the same high-impact question, the model gives conflicting advice about legal obligations, medical treatment, transfer of funds, safety operations, core facts or the authenticity of sources. Do not select the version with the most forceful tone, and do not keep asking until one appears that you prefer. Stop treating the answer as a basis for action, turn to an appropriate independent authority such as a regulator or original document, and consult an appropriately qualified professional where necessary. A model can help organise questions, list items that need checking or find material that needs reading. It cannot acquire the standing to replace external responsibility and professional judgement merely because it has repeated an answer several times.
The opposite error also needs to be avoided. Five matching answers may simply repeat the same error pattern, while five differing answers may be normal diversity in open-ended expression. Stability testing can reveal the scope of instability, but it cannot quantify real-world accuracy. Version records can help reproduce a process, but they cannot replace source verification. Their value is in turning “I feel that it changed” into conditions that can be checked, evidence that can be traced and limitations that can be explained.
Returning to the original question, different answers to the same question do not inherently mean that the model has failed, nor do they inherently mean that one passage is the correct answer. The dividing line is not whether two passages match word for word, but whether key conclusions recur stably under comparable conditions, whether the facts on which they rely can be traced and checked against external sources, and whether users have kept enough version and condition records when the answer can alter their actions. Treating an answer as a draft, lead or candidate plan that requires verification is the more reliable way to respond to difference.
