⭐ STELLAR

A Search-Based Testing Framework for Large Language Model Applications

🏆 IEEE Computer Society TCSE Distinguished Paper Award
33rd IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2026)

Architecture

The simplified test generation pipeline of STELLAR is as follows:

flowchart LR
    A[Feature Configs<br/>content/style/perturbation] --> B[Test Input Generator<br/>prompting + optional RAG]
    B --> C[SUT<br/>LLM app under test]
    C --> D[Evaluators<br/>judge LLM + similarity metrics]
    D --> E[Search Algorithm<br/>rs / gs / nsga2 / astral]
    E --> B
    D --> F[Results + Dashboard<br/>failures, rates, heatmaps]

    classDef config fill:#E3F2FD,stroke:#1565C0,stroke-width:2px,color:#000;
    classDef generator fill:#E8F5E9,stroke:#2E7D32,stroke-width:2px,color:#000;
    classDef sut fill:#FFF3E0,stroke:#EF6C00,stroke-width:2px,color:#000;
    classDef eval fill:#F3E5F5,stroke:#6A1B9A,stroke-width:2px,color:#000;
    classDef search fill:#FCE4EC,stroke:#C2185B,stroke-width:2px,color:#000;
    classDef results fill:#ECEFF1,stroke:#455A64,stroke-width:2px,color:#000;

    class A config;
    class B generator;
    class C sut;
    class D eval;
    class E search;
    class F results;

What each block does:

• Feature Configs: defines what kinds of inputs STELLAR is allowed to generate.
• Test Input Generator: creates candidate prompts/questions from those feature definitions.
• SUT: the system under test (for example, a Safety QA or Navigation QA application).
• Evaluators: score outputs and decide whether a test case is a failure.
• Search Algorithm: uses scores to decide which test cases to try next.
• Results + Dashboard: stores artifacts and helps inspect failures interactively.

Updates

• 2026-05-11: Added instructions for customization -> custom/README.md

• 2026-08-09: Added Jupyter notebooks and notebook guide -> jupyter/README.md

Overview

STELLAR is a search-based testing framework that automatically generates and runs test cases for LLM applications and identifies where the system fails.

It builds upon the OpenSBT infrastructure and uses Pymoo (v0.6.1.5) for search algorithms.

STELLAR helps to answer:

• Which user inputs are most likely to break my LLM application?
• How can I generate those inputs systematically instead of manually or randomly guessing them?
• How often does my application fail under different input styles and constraints?

Features

• ✅ Stylistic variation (e.g., implicitness, slang, politeness, anthropomorphism)
• ✅ Perturbation simulations (e.g., fillers, word deletions, homophones, typos)
• ✅ Content variation based on domain/category definitions
• ✅ Four generation algorithms (rs, nsga2, gs, astral)
• ✅ Automated result collection and reproducible experiment outputs
• ✅ Interactive dashboard for result exploration and failure analysis
• ✅ Weight and Biases Integration for experiment tracking

Jupyter Notebooks

For interactive walkthroughs and analysis, use the notebooks in jupyter/:

• Notebook Guide
• 01 Getting Started
• 02 Navigation
• 03 Safety
• 04 Dashboard

How It Works (At a Glance)

1. Pick an LLM application to test (for example, safety QA or navigation QA).
2. Define which input features you want to vary in a JSON config.
3. Run a search algorithm to generate and evaluate test cases.
4. Inspect failures and metrics in result files or the dashboard.

If You Are New to Optimization

You do not need deep optimization knowledge to run STELLAR.

• Think of an algorithm as a strategy for selecting the next test inputs.
• A population is just a batch of candidate test cases.
• A generation is one iteration where candidates are evaluated and improved.
• A fitness score indicates how strongly a test case exposes a target behavior (for example, unsafe or incorrect output).

For first runs, use the defaults in this README, then adjust one parameter at a time.

Project Structure

The project’s structure is outlined below as a high-level overview, omitting detailed files and scripts.

stellar/
│
├── analysis/            # Paper analysis scripts
├── configs/             # Feature config files
├── examples/            # Implementation of use cases Navi and Safety
├── custom/            # Tutorial and example to apply to custom problem
├── judge_eval/          # Files for the judge evaluation
├── llm/                 # Main folder extending OpenSBT to support LLM Testing
├── opensbt/             # OpenSBT Base Folder
├── .env-example         # Example .env file to use cloud LLMs
├── README.md            # Project overview
├── requirements.txt     # Dependencies
├── run_tests_navi.py    # Run navi case study
└── run_tests_safety.py  # Run safety case study

Installation

STELLAR requires Python to be installed and its compatibility has been tested with Python 3.11. STELLAR does not require GPU resources if cloud LLMs are used.

You can install dependencies via:

pip install -r requirements.txt

STELLAR can be used with local LLMs just as Llama3.2 or Mistral from Ollama, as well as with LLMs deployed in the cloud (OpenAI). Configure the OpenAI endpoint and key via .env.

When using local models, make sure that they have been downloaded via Ollama locally. Make sure hardware requirements are satisfied.

Getting Started

This framework integrates the following applications for testing:

• Standalone LLMs: Safety, Navigation Question Answering
• ConvNavi (RAG-based POI recommendations): Navigation Question Answering

The configuration for LLM related experiments is done via the config.py as well as directly by passing arguments via flags to a corresponding function.

Quick Start

If you are new to STELLAR, start with a small run first:

1. Choose one use case: Navigation or Safety.
2. Keep a small population and few generations (for example 5x5) so runs finish quickly.
3. Use a local model with Ollama, or configure OpenAI via .env.
4. Inspect the generated folder in results after execution.

After that, increase population size, generations, or runtime.

Navigation

Standalone LLM: run a simplified navigation recommendation example (using a local model here):

DEPLOYMENT_NAME="llama3.2" python run_tests_navi.py \
        --sut "IPA_LOS" \
        --judge "llama3.2" \
        --population_size 5 \
        --n_generations 5 \
        --algorithm "nsga2" \
        --max_time "00:10:00" \
        --features_config "configs/navi_features.json"\
        --use_repair \
        --no_wandb \
        --use_rag \
        --seed 1

RAG-based SUT: run a more advanced setup where ConvNavi (RAG-based) provides place recommendations. First set up ConvNavi, start it in server mode, then run:

DEPLOYMENT_NAME="llama3.2" python run_tests_navi.py \
        --sut "IPA_YELP" \
        --judge "llama3.2" \
        --population_size 5 \
        --n_generations 5 \
        --algorithm "nsga2" \
        --max_time "00:10:00" \
        --features_config "configs/navi_features.json"\
        --no_wandb \
        --use_rag \
        --seed 1

This run generates 25 test cases and stores outputs in the results folder.

Safety

To test how a standalone LLM handles malicious user inputs, run:

python run_tests_safety.py \
        --population_size 5 \
        --sut "llama3.2" \
        --judge "llama3.2" \
        --n_generations 5 \
        --algorithm nsga2 \
        --max_time "00:01:00" \
        --results_folder "/results/" \
        --features_config "configs/safety_features.json"\
        --seed 1

Search Configuration

STELLAR distinguishes between style, content, and perturbation features for test generation. You define these features in config files such as configs/navi_features.json. Modify these values to control how test inputs are generated.

At a high level, the algorithm controls how STELLAR chooses the next test cases. Different algorithms trade off speed, coverage, and ability to find subtle failures.

If you are unsure where to start:

1. Keep the feature set small.
2. Change only one feature or bound at a time.
3. Compare failure counts and failure types across runs.

Quick Algorithm Picker

Recommended first path: start with Random Search (rs) for a baseline, then switch to NSGA-II (nsga2) for deeper failure discovery.

Algorithms that exist in pymoo can be also used by implementing interfaces from OpenSBT.

Customization

To apply STELLAR to your custom problem follow tutorial in custom/README.md.

You can customize also operators and the testing definition as described in CUSTOMIZATION.

Wandb Integration

STELLAR integrates wandb for experiment progress monitoring and results tracking. Enable or disable wandb via the --wandb flag. Before logging, create a wandb project, log in with the CLI, and set the project name in the main application file. Result artifacts are uploaded to the corresponding run and can be downloaded for later analysis.

weave.init("dev")
wandb.init(
        entity="<your wandb group>",                  # team
        project="<your project name>",                  # the project name
        name=problem_name,                  # run name
        group=datetime.now().strftime("%d-%m-%Y"),  # group by date
        tags=tags,
)

Dashboard

STELLAR provides a Streamlit-based dashboard for interactive exploration of test results.

The dashboard allows you to:

• Browse results: Point to any results root folder and the dashboard recursively finds all completed runs.
• View summary metrics: Total test cases, number of failures, and overall failure rate.
• Failure distribution heatmaps: Select any two discrete input features (e.g., style, persuasion, category) and visualize failure rates as a heatmap.
• Filter utterances by cell: Pick specific feature value combinations to inspect the corresponding test cases.
• View all failures: Browse all critical (failure) test cases with questions, answers, and fitness scores.

Start the dashboard with:

streamlit run dashboard.py --server.headless true

Replication

timestamp=$(date +'%Y-%m-%d_%H-%M-%S')
base_output_dir="./judge_eval/out/session_${timestamp}"
for n in 1 3; do
    # Create parent folder: judge_eval/out/session_<timestamp>/sample-<n>-<agg>/
    technique_folder="${base_output_dir}/sample-${n}"
    mkdir -p "$technique_folder"

    for i in {1..6}; do
        # Create run subfolder
        run_folder="${technique_folder}/run${i}"
        mkdir -p "$run_folder"

        python -m judge_eval.nuanced_validation_dim \
            --models gpt-35-turbo DeepSeek-V3-0324 gpt-4o-mini gpt-4 gpt-4o gpt-5-chat mistral deepseek-v2 \
            --exp_name "sample-${n}-run${i}" \
            --dataset_path "<path to question answer pairs>" \
            --output_folder "$run_folder" \
            --n_questions 1000 \
            --n_samples $n \
            --aggregator "majority"
    done
done

#!/bin/bash
BASE_DIR="<path to the runs>"
GT_CSV="<path to aggregated human annotations>"
SAVE_DIR="./judge-eval/tmp"

mkdir -p "$SAVE_DIR"

# List of sample configurations and runs
SAMPLES=("sample-1-majority" "sample-3-majority")
RUNS=("run1" "run2" "run3" "run4" "run5" "run6")

for SAMPLE in "${SAMPLES[@]}"; do
  for RUN in "${RUNS[@]}"; do
    RUN_DIR="${BASE_DIR}/${SAMPLE}/${RUN}/${SAMPLE}-${RUN}"

    CSV_PATH=$(find "$RUN_DIR" -maxdepth 1 -type f -name "*validation-repeat*.csv" | head -n 1)

    if [[ -f "$CSV_PATH" ]]; then
      OUT_DIR="${SAVE_DIR}/${SAMPLE}/${RUN}"
      mkdir -p "$OUT_DIR"
      echo "Evaluating: ${SAMPLE} - ${RUN}"
      python -m analysis.rq0.evaluate_accuracy_judge_gt \
        --csv_path "$CSV_PATH" \
        --gt_csv_path "$GT_CSV" \
        --json_output_path "${OUT_DIR}/evaluation_results.json" \
        --plot_errors_file "${OUT_DIR}/prediction_errors.png" \
        --plot_efficiency_file "${OUT_DIR}/model_efficiency.png" \
        --plot_f1_file "${OUT_DIR}/model_f1_score.png"

      echo "Done."
    else
      echo "No validation-repeat CSV found in ${RUN_DIR}"
    fi
  done
done

bash analysis/rq0/run_statistical_test.sh

DATE=$(date +%d-%m-%Y)

# RANDOM
python run_tests_safety.py \
                --population_size 2000 \
                --n_generations 1 \
                --algorithm rs \
                --max_time "02:00:00" \
                --results_folder "/results/${DATE}/" \
                --features_config "configs/safety_features.json"\
                --seed 1

# T-wise
python run_tests_safety.py \
        --population_size 2000 \
        --n_generations 1 \
        --algorithm gs \
        --max_time "02:00:00" \
        --results_folder "/results/${DATE}/" \
        --features_config "configs/safety_features.json"\
        --seed 1

# STELLAR
python run_tests_safety.py \
        --population_size 20 \
        --n_generations 100 \
        --algorithm nsga2 \
        --max_time "02:00:00" \
        --results_folder "/results/${DATE}/" \
        --features_config "configs/safety_features.json"\
        --seed 1 \
        --use_repair

DATE=$(date +%d-%m-%Y)

# RANDOM
N_VALIDATORS=1 DEPLOYMENT_NAME="gpt-4o-mini" python run_tests_navi.py \
        --sut "IPA_YELP" \
        --population_size 10000 \
        --algorithm rs \
        --max_time "03:00:00" \
        --results_folder "/results/${DATE}/" \
        --features_config "configs/navi_features.json"\
        --no_wandb \
        --use_rag \
        --seed 1

# T-wise
N_VALIDATORS=1 DEPLOYMENT_NAME="gpt-4o-mini" python run_tests_navi.py \
        --sut "IPA_YELP" \
        --population_size 10000 \
        --algorithm gs \
        --max_time "00:30:00" \
        --results_folder "/results/${DATE}/" \
        --features_config "configs/navi_features.json"\
        --no_wandb \
        --use_rag \
        --seed 1

# STELLAR
N_VALIDATORS=1 DEPLOYMENT_NAME="gpt-4o-mini" python run_tests_navi.py \
        --sut "IPA_YELP" \
        --population_size 20 \
        --n_generations 30 \
        --algorithm "nsga2" \
        --max_time "03:00:00" \
        --results_folder "/results/${DATE}/" \
        --features_config "configs/navi_features.json"\
        --use_repair \
        --no_wandb \
        --use_rag \
        --seed 1

python -m analysis.rq12.get_analysis_safety

python -m analysis.rq12.get_analysis_navi

Citation

A preprint of the paper can be found on arXiv.

If you use our work in your research, if you extend it, or if you simply like it, please cite it in your publications.

Here is an example BibTeX entry:

@inproceedings{sorokin2026stellar,
  title     = {STELLAR: A Search-Based Testing Framework for Large Language Model Applications},
  author    = {Sorokin, Lev and Vasilev, Ivan and Friedl, Ken E. and Stocco, Andrea},
  booktitle = {Proceedings of the 33rd IEEE International Conference on Software Analysis, Evolution and Reengineering},
  year      = {2026},
  publisher = {IEEE},
  doi = {10.1109/SANER67736.2026.00093}
}

License

The software is distributed under MIT license. See the license file.

Authors

Lev Sorokin (lev.sorokin@tum.de)
Ivan Vasilev (ivan.vasilev@tum.de)