Wayfinder

A reference implementation for learning and building AI evaluation systems.

📖 Start the AI Evaluation Series • 💻 GitHub Releases • 🐛 Report an Issue

Wayfinder is a reference implementation for building and understanding AI evaluation systems.

It evolves a single AI-powered flight search application through rule-based evaluation, human evaluation, LLM-as-a-Judge, online evaluation, and experiment comparison.

The repository was built alongside the AI Engineering Fundamentals — AI Evaluation article series, with each stage introducing the next layer of the evaluation system.

What This Repository Covers

Software Tests vs AI Evals
        ↓
Evaluation Fundamentals
        ↓
Rule-Based Evaluation
        ↓
Human Evaluation
        ↓
LLM-as-a-Judge
        ↓
Online Evaluation
        ↓
Evaluation Experiments

Each stage builds on the same Wayfinder application, showing how an evaluation system can evolve as an AI application moves from development toward production.

Why Wayfinder?

Building an AI application is only the beginning.

The real engineering challenge is answering questions like:

• Is my AI application improving?
• Did my latest change introduce regressions?
• How do I evaluate subjective qualities like helpfulness or groundedness?
• How can I scale evaluations beyond manual review?
• How do I confidently ship AI applications to production?

Wayfinder explores these questions through practical implementations of modern AI evaluation techniques.

What's Implemented

• AI-powered flight search reference application
• Rule-Based Evaluation
• Human Evaluation
• LLM-as-a-Judge
• Online Evaluation
• Real user interaction tracing and explicit feedback
• Reference-free evaluation of production interactions
• Evaluation datasets and scoring rubrics
• LangSmith tracing and evaluation integration
• Evaluation experiment comparison
• Target and regression-guard evaluation cases
• Repeated evaluation runs

Project Structure

src/
└── wayfinder/
    ├── agent/
    ├── evaluators/
    ├── models/
    ├── services/
    └── tools/

examples/
├── wayfinder_cli.py
├── rule_based_evaluation/
├── human_evaluation/
├── llm_judge_evaluation/
├── online_evaluation/
└── experiment_comparison/

Prerequisites

• Python 3.12+
• uv

Clone the repository:

git clone https://github.com/DivakarUngatla/wayfinder.git
cd wayfinder

Install dependencies:

uv sync

Configuration

Copy the example environment file.

cp .env.example .env

OpenAI API Key

The interactive Wayfinder assistant and Human Evaluation examples use OpenAI models.

Create an API key:

https://platform.openai.com/api-keys

Then update your .env file.

OPENAI_API_KEY=your_api_key

Note

OpenAI is only required for examples that generate AI responses.

LangSmith

LangSmith is required for interaction tracing and the LangSmith-based evaluation examples, including Online Evaluation and Experiment Comparison.

Create an account and API key:

https://docs.langchain.com/langsmith/create-account-api-key

Then update your .env file.

LANGSMITH_API_KEY=your_api_key
LANGSMITH_PROJECT=wayfinder

Quick Start

Launch the AI Assistant

uv run python examples/wayfinder_cli.py

Interact with the AI assistant directly from your terminal.

Generate Human Evaluation Dataset

uv run python examples/human_evaluation/generate_responses.py

This generates representative evaluation samples containing:

• User query
• Expected behavior
• Assistant response
• Retrieved tool outputs

These samples can then be reviewed using the Human Evaluation workflow.

Run Rule-Based Evaluation

uv run python examples/rule_based_evaluation/local_evaluation.py

Run LangSmith Evaluation

uv run python examples/rule_based_evaluation/langsmith_evaluation.py

Run LLM-as-a-Judge Evaluation

uv run python examples/llm_judge_evaluation/local_evaluation.py

Run LLM-as-a-Judge Evaluation with LangSmith

uv run python examples/llm_judge_evaluation/langsmith_evaluation.py

Run Online Evaluation

First, interact with Wayfinder to generate traced user interactions:

uv run python examples/wayfinder_cli.py

Then run the online evaluator:

uv run python examples/online_evaluation/evaluate_recent_runs.py

The evaluator processes recent Wayfinder interactions captured in LangSmith and attaches automated quality scores and explanations back to each trace.

Explicit user feedback collected through the CLI is also attached to the corresponding LangSmith trace.

Run an Evaluation Experiment

Run the focused experiment-comparison dataset against the current version of Wayfinder:

uv run python examples/experiment_comparison/compare_experiments.py \
  --dataset-name wayfinder_experiment_comparison \
  --prefix wayfinder-experiment \
  --repetitions 5

The runner evaluates the same examples repeatedly and records the results as a LangSmith experiment. Run it against different application versions to compare behavior using the same dataset and evaluator.

AI Engineering Fundamentals

Wayfinder was built alongside the AI Engineering Fundamentals — AI Evaluation article series.

Each article introduces an AI evaluation concept, while this repository provides the runnable implementation.

Foundations

• ✅ Part 1 — Software Tests vs AI Evals
• ✅ Part 2 — Understanding AI Evaluation

Evaluation Techniques

• ✅ Part 3 — Rule-Based Evaluation
• ✅ Part 4 — Human Evaluation
• ✅ Part 5 — LLM-as-a-Judge
• ✅ Part 6 — Online Evaluation
• ✅ Part 7 — Comparing Evaluation Experiments

Releases

Each GitHub release captures a reproducible milestone in Wayfinder's evolution.

Earlier releases correspond to the implementation developed in each article. For Part 7, v0.6.0 provides the baseline used for the experiment comparison, while main contains the completed candidate implementation.

Contributing

Contributions, ideas, bug reports, and suggestions are always welcome.

If you'd like to improve Wayfinder or discuss AI evaluation techniques, feel free to open an issue or submit a pull request.

License

This project is licensed under the MIT License.