> ## Documentation Index
> Fetch the complete documentation index at: https://arizeai-433a7140-claude-llms-txt-2026-08-12.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Pydantic Evals

> How to use Pydantic Evals with Phoenix to evaluate AI applications using structured evaluation frameworks

[Pydantic Evals](https://github.com/pydantic/pydantic-evals) is an evaluation library that provides preset direct evaluations and LLM Judge evaluations. It can be used to run evaluations over dataframes of cases defined with Pydantic models. This guide shows you how to use Pydantic Evals alongside Arize Phoenix to run evaluations on traces captured from your running application.

## Launch Phoenix

<Card>
  <Tabs>
    <Tab title="Self-Host">
      Run Phoenix on your own infrastructure, backed by PostgreSQL so traces persist beyond a single process. This is the option to reach for once Phoenix is shared across a team or environment.

      The [self-hosting guide](/docs/phoenix/self-hosting) covers [Kubernetes](/docs/phoenix/self-hosting/deployment-options/kubernetes), [Helm](/docs/phoenix/self-hosting/deployment-options/kubernetes-helm), [Railway](/docs/phoenix/self-hosting/deployment-options/railway), [AWS CloudFormation](/docs/phoenix/self-hosting/deployment-options/aws-with-cloudformation), [Google Cloud Run](/docs/phoenix/self-hosting/deployment-options/google-cloud-run), [Azure](/docs/phoenix/self-hosting/deployment-options/azure), and [Render](/docs/phoenix/self-hosting/deployment-options/render), plus authentication and configuration.
    </Tab>

    <Tab title="Local">
      ```bash theme={null}
      uvx arize-phoenix serve
      ```

      No [uv](https://docs.astral.sh/uv/)? `pip install arize-phoenix && phoenix serve` does the same thing. See [Terminal setup](/docs/phoenix/environments#terminal) for customization.
    </Tab>

    <Tab title="Container">
      ```bash theme={null}
      docker run -p 6006:6006 -p 4317:4317 arizephoenix/phoenix:latest
      ```

      Images are published to [Docker Hub](https://hub.docker.com/r/arizephoenix/phoenix). See [Docker](/docs/phoenix/self-hosting/deployment-options/docker) for volumes, PostgreSQL, and other options.
    </Tab>
  </Tabs>

  Phoenix serves its UI and OTLP HTTP on port **6006**, and OTLP gRPC on port **4317**. For a local instance that's [http://localhost:6006](http://localhost:6006) — leave it running while you work.
</Card>

**Install packages:**

```bash theme={null}
pip install arize-phoenix-otel
```

Point your code at the Phoenix instance you started. The endpoint below is the default for a local `phoenix serve`; for a deployment running elsewhere, use its hostname instead.

```python theme={null}
import os

os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"

# Only if the deployment has authentication enabled
# os.environ["PHOENIX_API_KEY"] = "your-api-key"
```

## Install

```sh theme={null}
pip install pydantic-evals arize-phoenix openai openinference-instrumentation-openai
```

## Setup

Enable Phoenix tracing to capture traces from your application:

```python theme={null}
from phoenix.otel import register

tracer_provider = register(
    project_name="pydantic-evals-tutorial",
    auto_instrument=True,  # Automatically instrument OpenAI calls
)
```

## Basic Usage

### 1. Generate Traces to Evaluate

First, create some example traces by running your AI application. Here's a simple example:

```python expandable theme={null}
from openai import OpenAI
import os

client = OpenAI()

inputs = [
    "What is the capital of France?",
    "Who wrote Romeo and Juliet?",
    "What is the largest planet in our solar system?",
]

def generate_trace(input):
    client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "You are a helpful assistant. Only respond with the answer to the question as a single word or proper noun.",
            },
            {"role": "user", "content": input},
        ],
    )

for input in inputs:
    generate_trace(input)
```

### 2. Export Traces from Phoenix

Export the traces you want to evaluate:

```python theme={null}
import phoenix as px
from phoenix.client.types.spans import SpanQuery

query = SpanQuery().select(
    "llm.input_messages",
    "llm.output_messages",
)

# Query spans from Phoenix
from phoenix.client import Client
client = Client()
spans = client.spans.get_spans_dataframe(query=query, project_name="pydantic-evals-tutorial")
spans = spans.rename(columns={"llm.input_messages": "input", "llm.output_messages": "output"})
spans["input"] = spans["input"].apply(lambda x: x[1].get("message").get("content"))
spans["output"] = spans["output"].apply(lambda x: x[0].get("message").get("content"))
```

### 3. Define Evaluation Dataset

Create a dataset of test cases using Pydantic Evals:

```python expandable theme={null}
from pydantic_evals import Case, Dataset

cases = [
    Case(
        name="capital of France",
        inputs="What is the capital of France?",
        expected_output="Paris"
    ),
    Case(
        name="author of Romeo and Juliet",
        inputs="Who wrote Romeo and Juliet?",
        expected_output="William Shakespeare",
    ),
    Case(
        name="largest planet",
        inputs="What is the largest planet in our solar system?",
        expected_output="Jupiter",
    ),
]
```

### 4. Create Custom Evaluators

Define evaluators to assess your model's performance:

```python expandable theme={null}
from pydantic_evals.evaluators import Evaluator, EvaluatorContext

class MatchesExpectedOutput(Evaluator[str, str]):
    def evaluate(self, ctx: EvaluatorContext[str, str]) -> float:
        is_correct = ctx.expected_output == ctx.output
        return is_correct

class FuzzyMatchesOutput(Evaluator[str, str]):
    def evaluate(self, ctx: EvaluatorContext[str, str]) -> float:
        from difflib import SequenceMatcher

        def similarity_ratio(a, b):
            return SequenceMatcher(None, a, b).ratio()

        # Consider it correct if similarity is above 0.8 (80%)
        is_correct = similarity_ratio(ctx.expected_output, ctx.output) > 0.8
        return is_correct
```

### 5. Setup Task and Dataset

Create a task that retrieves outputs from your traced data:

```python theme={null}
import nest_asyncio
nest_asyncio.apply()

async def task(input: str) -> str:
    output = spans[spans["input"] == input]["output"].values[0]
    return output

# Create dataset with evaluators
dataset = Dataset(
    cases=cases,
    evaluators=[MatchesExpectedOutput(), FuzzyMatchesOutput()],
)
```

### 6. Add LLM Judge Evaluator

For more sophisticated evaluation, add an LLM judge:

```python theme={null}
from pydantic_evals.evaluators import LLMJudge

dataset.add_evaluator(
    LLMJudge(
        rubric="Output and Expected Output should represent the same answer, even if the text doesn't match exactly",
        include_input=True,
        model="openai:gpt-4o-mini",
    ),
)
```

### 7. Run Evaluation

Execute the evaluation:

```python theme={null}
report = dataset.evaluate_sync(task)
print(report)
```

## Advanced Usage

### Upload Results to Phoenix

Upload your evaluation results back to Phoenix for visualization:

```python expandable theme={null}
# Extract results from the report
results = report.model_dump()

# Create dataframes for each evaluator
meo_spans = spans.copy()
fuzzy_label_spans = spans.copy()
llm_label_spans = spans.copy()

for case in results.get("cases"):
    # Extract evaluation results
    meo_label = case.get("assertions").get("MatchesExpectedOutput").get("value")
    fuzzy_label = case.get("assertions").get("FuzzyMatchesOutput").get("value")
    llm_label = case.get("assertions").get("LLMJudge").get("value")

    input = case.get("inputs")

    # Update labels in dataframes
    meo_spans.loc[meo_spans["input"] == input, "label"] = str(meo_label)
    fuzzy_label_spans.loc[fuzzy_label_spans["input"] == input, "label"] = str(fuzzy_label)
    llm_label_spans.loc[llm_label_spans["input"] == input, "label"] = str(llm_label)

# Add scores for Phoenix metrics
meo_spans["score"] = meo_spans["label"].apply(lambda x: 1 if x == "True" else 0)
fuzzy_label_spans["score"] = fuzzy_label_spans["label"].apply(lambda x: 1 if x == "True" else 0)
llm_label_spans["score"] = llm_label_spans["label"].apply(lambda x: 1 if x == "True" else 0)

# Upload to Phoenix
from phoenix.client import Client
client = Client()
client.spans.log_span_annotations_dataframe(dataframe=meo_spans, annotation_name="Direct Match Eval", annotator_kind="CODE")
client.spans.log_span_annotations_dataframe(dataframe=fuzzy_label_spans, annotation_name="Fuzzy Match Eval", annotator_kind="CODE")
client.spans.log_span_annotations_dataframe(dataframe=llm_label_spans, annotation_name="LLM Match Eval", annotator_kind="LLM")
```

### Custom Evaluation Workflows

You can create more complex evaluation workflows by combining multiple evaluators:

```python expandable theme={null}
from pydantic_evals.evaluators import Evaluator, EvaluatorContext
from typing import Dict, Any

class ComprehensiveEvaluator(Evaluator[str, str]):
    def evaluate(self, ctx: EvaluatorContext[str, str]) -> Dict[str, Any]:
        # Multiple evaluation criteria
        exact_match = ctx.expected_output == ctx.output

        # Length similarity
        length_ratio = min(len(ctx.output), len(ctx.expected_output)) / max(len(ctx.output), len(ctx.expected_output))

        # Semantic similarity (simplified)
        from difflib import SequenceMatcher
        semantic_score = SequenceMatcher(None, ctx.expected_output.lower(), ctx.output.lower()).ratio()

        return {
            "exact_match": exact_match,
            "length_similarity": length_ratio,
            "semantic_similarity": semantic_score,
            "overall_score": (exact_match * 0.5) + (semantic_score * 0.3) + (length_ratio * 0.2)
        }
```

## Observe

Once you have evaluation results uploaded to Phoenix, you can:

* **View evaluation metrics**: See overall performance across different evaluation criteria

* **Analyze individual cases**: Drill down into specific examples that passed or failed

* **Compare evaluators**: Understand how different evaluation methods perform

* **Track improvements**: Monitor evaluation scores over time as you improve your application

* **Debug failures**: Identify patterns in failed evaluations to guide improvements

The Phoenix UI will display your evaluation results with detailed breakdowns, making it easy to understand your AI application's performance and identify areas for improvement.

## Resources

* [Pydantic Evals Documentation](https://github.com/pydantic/pydantic-evals)

* [Phoenix Evaluation Guide](/docs/phoenix/evaluation/evals)

* [Pydantic Evals Tutorial Notebook](https://github.com/Arize-ai/tutorials/blob/main/python/cookbooks/phoenix_evals_examples/pydantic-evals.ipynb)