Summarization

The Summarization scorer is a default LLM judge scorer that measures whether your LLM can accurately summarize text. In this case, the actual_output is the summary, and the input is the text to summarize.

Required Fields

To run the Summarization scorer, you must include the following fields in your Example:

• input
• actual_output

Scorer Breakdown

Summarization scores are calculated by determining:

1. Whether the summary contains contradictory information from the original text.
2. Whether the summary contains all of the important information from the original text.

To do so, we compute two subscores respectively:

$$\text{contradiction score} = \frac{\text{Number of Contradictory Statements in Summary}}{\text{Total Number of Statements in Summary}}$$

For the information score, we generate a list of important questions from the original text and check the fraction of the questions that are answered by information inthe summary.

$$\text{information score} = \frac{\text{Number of Important Questions Answered in Summary}}{\text{Total Number of Important Questions}}$$

The final score is the minimum of the two subscores.

$$\text{Summarization Score} = \min(\text{contradiction score}, \text{information score})$$

Sample Implementation

summarization.py

from judgeval import JudgmentClient
from judgeval.data import Example
from judgeval.scorers import SummarizationScorer

client = JudgmentClient()
example = Example(
    input="...",
    # Replace this with your LLM system's summary
    actual_output="...",
)
# supply your own threshold
scorer = SummarizationScorer(threshold=0.8)

results = client.run_evaluation(
    examples=[example],
    scorers=[scorer],
    model="gpt-4o",
)
print(results)