An LLM confidence score is something developers often look for when building a support-question agent—deciding whether to send an answer straight to the user or hand it over to a human. However, a number obtained simply by asking the model how sure it is cannot be trusted at face value. Below we examine why that happens and what calibration means.

Summary

A number produced when you ask an LLM “How sure are you?” is not a measurement from an internal gauge, but a sequence of tokens generated just like any other sentence. Asking the same question again or changing the wording changes the number. A trustworthy confidence score requires calibration—recording predictions and checking how often answers with a given score actually turned out to be right.

What is an LLM confidence score, and what does it mean to be “calibrated”?

One-line answer: A confidence score is calibrated when the probability attached to a prediction matches how often predictions with that score actually turn out to be correct across many cases.

A forecaster’s confidence comes from information—today’s measurements combined with past records. Given the same information and the same rule, the forecaster produces the same confidence. Whether a confidence score is accurate cannot be judged from a single case; you have to group all cases where the same confidence was given and compare that score with how often the event actually happened. If the actual frequency matches the stated confidence, the confidence is calibrated. If the event happened less often than stated, the forecaster is overconfident; if it happened more often, underconfident.

Calibration alone, however, is not enough. A forecaster who always repeats the seasonal average every single day is also calibrated over the long run, yet tells you nothing useful about a specific day. Useful confidence both uses observable information about the case at hand and is calibrated. Moreover, someone must record every prediction alongside the actual outcome; without that record, nobody can know whether a confidence score is accurate.

In their ICML 2017 paper “On Calibration of Modern Neural Networks”, Guo, Pleiss, Sun, and Weinberger define calibration as the property that the probability attached to a prediction reflects its actual likelihood of being correct. Under perfect calibration, among all predictions assigned confidence p, the fraction that turns out to be correct equals p. A reliability diagram visualizes this by grouping predictions into confidence bins and plotting accuracy as a function of confidence. A perfectly calibrated model lies on the diagonal, while any deviation from the diagonal represents miscalibration. The paper also reports that modern, deeper and larger neural networks tend to be overconfident, assigning higher confidence than their actual accuracy warrants.

Reliability diagram: actual accuracy by confidence bin compared with the perfect-calibration diagonal

Why is a confidence score stated by an LLM hard to trust?

One-line answer: Because an LLM generates text one token at a time based on preceding tokens rather than reading a value from an internal gauge.

An LLM writes text one token at a time based on previous tokens, a process known as autoregressive generation. For background on how models split and generate text token by token, see What are tokens and the context window? How LLMs count text and why length limits exist. When a model outputs a stated confidence score in text, that number is written the exact same way as any other word—as a plausible continuation of the sequence, not a reading from an internal gauge. Pretraining scores the model on predicting the next token of text, not on whether its stated confidence matches how often it is right.

This produces three observable consequences:

  • Asking the same question again can change the number.
  • Changing the wording—such as asking “Are you sure?”—can change the number.
  • Text already written before the number (for example, a sentence asserting that the answer is clearly right) pulls the subsequent confidence number along with it.

These points were laid out in the blog post “Asking LLM for confidence” and debated in a Hacker News thread titled “Using LLM for confidence scoring is a bad idea”. In that discussion, some commenters suggested using rubrics or small scales and others pointed to token probabilities, while other participants noted that those approaches still require calibration against actual outcomes.

How do you build a trustworthy confidence score instead?

One-line answer: You build an external rule using observable information, record every answer alongside its actual outcome, and update the rule based on per-group accuracy.

A prompt that simply asks the model how sure it is cannot bypass this limitation; for broader context on prompt design, see our prompt engineering notes (Korean). Using a support-question agent as an example scenario, the blog post outlines five steps for building a confidence rule outside the LLM:

  1. Choose observable information: pick signals you can check outside the model—such as whether search found a relevant document, whether this type of question has been answered before, or whether two answers generated for the same question agree.
  2. Start from the past base rate: use the historical overall correctness rate as your starting point.
  3. Record every answer and outcome: log every answer, its observable information, the confidence assigned to it, and whether it actually turned out to be correct.
  4. Update the rule from per-group correctness: group cases by their observable information and update the scoring rule using the actual correctness rate of each group.
  5. Re-check periodically and fix drift: keep checking the groups over time and adjust the rule whenever predictions become overconfident or underconfident.

Five-step confidence rule loop for a support-question agent

Ultimately, the rule is only as good as the record backing it.

How is this different from token probabilities (logprobs)?

One-line answer: Token probabilities reflect the model’s internal probability for the next token rather than words written in text, but they still require calibration against recorded outcomes.

There is an important distinction between a confidence score written out in words and the model’s internal next-token probability (logprob). Token probabilities are closer to something measured inside the model rather than a conversational continuation. Even so, the probability assigned to the next token is not automatically the probability that the overall answer is factually correct, so token probabilities also need a calibration check against recorded outcomes.

Looking inside the model’s internal mechanisms is a different approach covered in What is Mechanistic Interpretability?, and sampling settings are covered in What are Temperature and Top-p? Why LLM answers change every time.

What are the frequently asked questions?

One-line answer: Changing the prompt alone does not guarantee calibration; a useful score must use case-specific information and be checked against accumulated records.

Does changing the prompt make the stated confidence trustworthy?

No, not by itself. Even if you rephrase the prompt or provide a rubric, you still have to check the resulting scores against recorded outcomes to know whether they match actual correctness.

Is a confidence score good as long as it is calibrated?

No. A forecaster who always outputs the seasonal average is calibrated overall, but provides no useful information about a specific case. A good confidence score both uses observable case information and is calibrated.

What should you do when you have very few records?

Start from the past base rate. Your groups will be rough at first, and you can refine the rule as you record more answers and their actual outcomes.

One-line answer: These related posts cover token generation, prompt engineering, mechanistic interpretability, and sampling settings.

What are the sources?

One-line answer: The original blog post, the Hacker News discussion, and the ICML 2017 paper on neural network calibration.