Running LLM evaluations with OpenAI Evals

Evaluations (“evals”) are repeatable tests that measure whether an LLM output meets your quality bar for a specific task. They help you compare models, prompts, and code changes using the same dataset and scoring, so you can spot regressions early instead of relying on vibes. Evals are important because LLM behavior is non-deterministic and can drift with small changes—without a harness, it’s easy to ship breakage you only notice in production.

OpenAI Evals can look like a full framework when you first open it. In practice, most evaluations come down to a small set of moving parts.

• You register an eval name in a YAML file.
• You point it at a JSONL dataset.
• You choose a scoring approach, basic exact match, JSON match, or model graded.

Once you have that mental model, the rest is configuration.

This post uses a real world example, a resume extraction eval called cv-extraction. The goal is straightforward. Take messy CV text, produce JSON, and preserve content. No summarization, no dropped bullet points, no silent rewriting. It is a practical test case because CVs are long, formatting is messy, and models tend to drop details unless you force them not to.

Step 1, clone and install

From a clean machine:

git clone https://github.com/openai/evals.git
cd evals
pip install -e .

Then set your API key:

export OPENAI_API_KEY="sk-your-api-key-here"

If you prefer .env, you can use that too:

echo "OPENAI_API_KEY=sk-your-api-key-here" > .env

Quick sanity check that everything is working:

oaieval gpt-3.5-turbo test-match

If that runs, you are set.

Step 2, understand the three files that matter

For cv-extraction, there are three files that make the eval work. Everything else is optional.

• Eval definition: evals/registry/evals/cv-extraction.yaml
• Dataset: evals/registry/data/cv-extraction/samples.jsonl
• Model grader spec: evals/registry/modelgraded/cv-extraction.yaml

How they fit together:

• The eval definition names an eval, selects an eval class, and points to the dataset.
• The dataset provides per sample fields like input and ideal.
• The model grader spec is only used for model graded evals. It defines the grading prompt and which grades are valid.

Quick samples, what the files actually look like

These are trimmed down versions of the real files, just enough to show the structure.

evals/registry/evals/cv-extraction.yaml (the list of runnable evals)

cv-extraction:
  id: cv-extraction.v1
  description: Evaluate CV to JSON extraction with lossless preservation

cv-extraction.json-match:
  class: evals.elsuite.basic.json_match:JsonMatch
  args:
    samples_jsonl: cv-extraction/samples.jsonl

cv-extraction.model-graded:
  class: evals.elsuite.modelgraded.classify:ModelBasedClassify
  args:
    samples_jsonl: cv-extraction/samples.jsonl
    eval_type: cot_classify
    modelgraded_spec: cv-extraction

What to notice:

• class selects the eval template.
• samples_jsonl is relative to evals/registry/data/.
• modelgraded_spec is a registry key, not a file path.

evals/registry/data/cv-extraction/samples.jsonl (one JSON object per line)

{"input":[{"role":"system","content":"You are a CV-to-JSON transcription assistant. NEVER DELETE CONTENT."},{"role":"user","content":"Sylvester Asare Sarpong, truncated"}],"ideal":"{ \"basics\": {\"name\": \"Sylvester Asare Sarpong\"}, \"work\": [{\"company\": \"Example\"}], \"skills\": [\"Example\"] }"}

What to notice:

• input is chat format, a list of role and content messages.
• ideal is the expected JSON output, stored as a string.

evals/registry/modelgraded/cv-extraction.yaml (the grading rubric)

cv-extraction:
  choice_strings: ["A", "B", "C", "D", "F"]
  choice_scores:
    A: 1.0
    B: 0.9
    C: 0.7
    D: 0.5
    F: 0.0
  input_outputs:
    input: output
  prompt: |
    You are evaluating a CV-to-JSON extraction system.
    INPUT CV:
    {input}
    EXPECTED OUTPUT (ideal):
    {ideal}
    ACTUAL OUTPUT (from system):
    {output}
  eval_type: cot_classify

What to notice:

• The prompt uses {input}, {ideal}, {output} placeholders.
• input_outputs is the wiring that makes {output} available to the grader prompt.

Step 3, the eval registry YAML, what to look at

Open evals/registry/evals/cv-extraction.yaml. You will typically see runnable entries like:

• cv-extraction.basic
• cv-extraction.json-match
• cv-extraction.model-graded

The key fields in each entry:

• class: which template you are using.
  • evals.elsuite.basic.match:Match checks string match.
  • evals.elsuite.basic.json_match:JsonMatch checks JSON equality, keys and values, not whitespace or key order.
  • evals.elsuite.modelgraded.classify:ModelBasedClassify uses an LLM to grade the model output.
• args.samples_jsonl: where the dataset lives, relative to evals/registry/data/.
• args.modelgraded_spec: only for model graded, this points to a named entry in evals/registry/modelgraded/*.yaml.
• args.eval_type: how the grader formats its response. cot_classify is a good default because it allows the grader to reason and then choose.

Practical advice. Start with json-match. If your model does not reliably emit valid JSON, model grading will mostly measure formatting failures.

Step 4, the dataset JSONL, what each line needs

Open evals/registry/data/cv-extraction/samples.jsonl.

Each line is one test case. For this eval it includes:

• input: a chat style prompt. It typically includes a system message with strict instructions and a user message containing the raw CV.
• ideal: the gold JSON you’d like the model to produce.

That is enough to run:

• Match and JsonMatch compare the model output against ideal.
• ModelBasedClassify still uses ideal, but as reference material inside the grading prompt.

One detail people miss. Your input can be full chat format, a list of {role, content} objects. Evals supports this, and it keeps prompts easier to maintain.

Step 5, the model graded YAML, what matters and why

Open evals/registry/modelgraded/cv-extraction.yaml. This is the spec named cv-extraction. It is used because the eval definition includes modelgraded_spec: cv-extraction.

Key sections:

• choice_strings: the only valid grades, here it is A/B/C/D/F.
• choice_scores: map those grades to numbers so you get a clean overall score.
• input_outputs: the wiring. In this case:
  • run the policy model on input.
  • store its completion under output.
  • reference {output} inside the grader prompt.
• prompt: the rubric the grader uses. It references {input}, {ideal}, and {output}.
• eval_type: cot_classify asks the grader to place the final grade at the end so it is easy to parse.

The prompt should be specific. Avoid a vague question like "is this good". Call out concrete failure modes like missing bullets, dropped skills, collapsed sections, formatting drift, and missing project tech stacks.

Step 6, run the evals

Run basic match:

oaieval gpt-3.5-turbo cv-extraction.basic

Run JSON match:

oaieval gpt-3.5-turbo cv-extraction.json-match

Run model-graded:

oaieval gpt-3.5-turbo cv-extraction.model-graded

Practical note. Model graded evals cost more. You call a model to produce JSON, then call another model to grade it.

Step 7, where results go, and what to check first

By default, results are logged locally as JSONL under a record path like /tmp/evallogs.

What I check first:

• The raw model outputs: whether the system emits valid JSON, or leaks markdown and prose.
• Failure mode patterns: missing bullets, merged bullets, dropped tech stacks, collapsed paragraphs.
• Model graded distribution: a cluster of __invalid__ or strange grades usually means the grader prompt format is too loose.

Step 8, the add a new eval playbook

If you are creating your own eval, do it in this order:

• Write 3 to 20 samples first in evals/registry/data/<your_eval>/samples.jsonl.
• Pick a template:
  • Match if the output is basically fixed.
  • JsonMatch if the output is strict JSON and you care about exact structure.
  • ModelBasedClassify if the output is flexible or you care about nuanced rubrics.
• Register the eval in evals/registry/evals/<your_eval>.yaml.
• If model graded, add a named entry in evals/registry/modelgraded/<your_eval>.yaml.

Avoid starting with hundreds of samples. You will iterate on the prompt and schema, and you will learn faster with a small, high signal set.

Bonus, a gotcha with base eval aliases

Registry YAMLs sometimes include a base alias like:

• cv-extraction: { id: cv-extraction.v1 }

That is meant to point at a real runnable eval entry. If the id points to a name that does not exist, oaieval <model> cv-extraction will fail even if cv-extraction.model-graded works.

When in doubt, run the explicit eval entry you want, like cv-extraction.model-graded.

If you’re stuck, here are the three usual culprits

• You’re editing the wrong file: the CLI uses the registry, not random YAML in the data directory.
• Your JSONL keys don’t match the prompt: the grader prompt references {input} and {ideal}, so the dataset needs those keys.
• Your model output isn’t parseable: fix JSON generation first, then worry about grading nuance.

That’s basically it. Once you’ve got one eval working, adding more is just more samples and better rubrics.