JSONCorrectness

The JSONCorrectness scorer is a default scorer that checks whether your LLM’s actual_output matches your JSON schema.

Required Fields

To run the JSONCorrectness scorer, you must include the following fields in your Example:

• input
• actual_output

Scorer Breakdown

JSONCorrectness scores are calculated with a binary score representing whether the actual_output matches the JSON schema.

To define a JSON schema, you can define a pydantic BaseModel and pass it to the JSONCorrectness scorer.

example_schema.py

from pydantic import BaseModel

class SampleSchema(BaseModel):
    field1: str
    field2: int

$$\text{JSONCorrectness} = \begin{cases} 1 & \text{if } \text{actual output} \text{ matches } \text{schema} \\ 0 & \text{otherwise} \end{cases}$$

Sample Implementation

json_correctness.py

from judgeval import JudgmentClient
from judgeval.data import Example
from judgeval.scorers import JSONCorrectnessScorer
client = JudgmentClient()
example = Example(
    input="Create a JSON object with the keys 'field1' (str) and 'field2' (int). Fill them with random values.",
    # Replace this with your LLM system's output
    actual_output="{'field1': 'value1', 'field2': 1}",
)
scorer = JSONCorrectnessScorer(threshold=0.8)
results = client.run_evaluation(
    examples=[example],
    scorers=[scorer],
    model="gpt-4o",
)
print(results)