G-Eval ^[1] is in 2026 one of the most widely recommended recipes for setting up LLM-as-a-judge. It was introduced back in 2023. Today, many open-source evaluation libraries ship a ready-made implementation of it. Because it is so often the default choice, it is worth understanding what it actually does, what the original paper did and did not show, and where this method fits in the evaluation framework of the previous sections.

In short: G-Eval allows us to generate numerical evaluation scores for texts, using LLM as a judge frameowrk. The appeal of an LLM judge method is that it is reference-free: you describe what good looks like, and the model judges the output directly.

This makes the method a natural candidate when setting up an evaluation metric in a rubric. We don't need to let the LLM generate different output labels, and convert these labels to numerical scores afterwards. G-Eval does this both in one go.

How does it work?

The recipe for G-Eval is short.

1. The judge prompt. For G-Eval we write a judge prompt that contains a task introduction (what is being evaluated) and the evaluation criterion (what counts as good, and on what scale).

2. Auto chain-of-thought. We then ask an LLM to turn these instructions into an evaluation form, together with a list of more detailed instructions. The final step is always to assign a score on the given scale. These generated steps, together with the earlier specified instructions then form our final fixed judge prompt. This is done once up front, during the implementation. We don't write a new chain of thought for every output.

3. The scoring function. The fixed prompt, the input context and the output to be evaluated are then sent to the LLM, which is asked to fill in a simple form with the score that the LLM assigns. But rather than using the score the LLM has generated as output (which doesn't have to be reliable), the scoring function looks at the probabilities the LLM assigns to each possible score token, and computes the expected value:

$\text{score} = \sum_{i=1}^{n} p(s_i) \times s_i$

where $s_1, \dots, s_n$ are the allowed scores (e.g. 1 to 5) and $p(s_i)$ is the probability the LLM assigns to $s_i$.

Rather than letting the LLM generates itself generate a label and converting this to a numerical value, we look at the probabilities of the tokens, and use those as a measure for how 'certain' the LLM is about different scores, to directly convet the assigned labels to a more quantitative output in one go.

Suppose the judge evaluates two responses:

If you simply took the most likely answer, both responses would get a 3 and be indistinguishable. The probability-weighted score shows that the judge leaned noticeably more towards "4" for response A.

Why does it work?

Before G-Eval, earlier LLM-based evaluators, such as GPTScore ^[2], scored the quality of a text based on the probabilities that an LLM would assign to the generated text itself. The idea was to use the probability distribution for the next token that the LLM has learned, as a measure of the quality of a piece of text. Working from the underlying assumption that an LLMs has learned to assign higher probabilities to high-quality, fluent text.

The extra step within GEval is to not ask how probable a text itself is. But to first use the judge prompt to let the LLM generate series of verdict scores, and then looks at the probable the LLM has on each of these score. Think of it as a measure of how 'certain' the LLM is about its own verdicts.

Averaging over the probabilities associated to the generated scores fixes two known issues that come up when using LLMs to directly assign evaluation scores.

1. One value dominates. On a 1–5 scale, most LLMs tends to answer the same middle value (typically 3) for lots of inputs. The scores then have very little variance, and do not correlate well with human ratings.
2. Ties everywhere. LLMs almost always answer with an integer, even if you explicitly ask for decimals. With only five possible values across hundreds of outputs, many outputs get exactly the same score, and the subtle differences between them are lost.

The probability-weighted sum fixes both: it turns a coarse integer label into a continuous score that preserves the judge's uncertainty. Think of it as a measure of how 'certain' the LLM is about its own verdict, folded into the score itself.

Because this method only needs a written criterion, it can be easily generalised to evaluate almost any general quality of a text. The original paper tested this method on coherence, consistency, fluency, and relevance of summaries, and the naturalness, coherence, engagingness and groundedness of dialogue responses. This makes G-Eval a natural candidate for many qualitative metrics, such as the formal tone of voice metric $S_1$ in our earlier example rubric, which are hard to capture in rule-based checks.

How well does it work?

But being applicable to any criterion is not the same as working well for it. In the original paper, the reliability of the metric is measured by looking how well the output scores correlate with human ratings. For every new criterion, the agreement with SMEs will depend on the details of the judge prompt, dataset, and the LLM-model used, and has to be measured again in order to know how reliable this is.

The original paper also flags a problem that G-Eval shares with other LLM-based metrics. Because the LLM's probabilities are used as a measure of the quality of a piece of text, this method can be biased to prefer LLM-written text (that follows exactly the probabilistic text patterns than an LLM would expect). Hence, this evalaution method may prefer LLM-written text over human-written text.

The GPT-4 judge gave summaries written by GPT-3.5 higher scores than summaries written by freelance writers, even where human annotators preferred the human-written ones. (Note that the judge and the generator were different models here). LLM models from the same family may share the same bias on what a 'good' piece of text looks like.