Prompt Ops

🎉 New: Prompt Duel Optimizer (PDO) Published!

We've published a new paper on PDO (Prompt Duel Optimizer) - an efficient label-free prompt optimization method using dueling bandits and Thompson sampling. PDO achieves state-of-the-art results on BIG-bench Hard and MS MARCO benchmarks.

📄 Read the paper: LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimization (arXiv:2510.13907)

🧪 Try it yourself: Check out the Web of Lies use case demonstrating PDO on logical reasoning tasks

⭐ Star this repo and follow along - we'll be publishing a detailed tutorial notebook soon!

---

What is prompt-ops?

prompt-ops is a Python package that automatically optimizes prompts for Llama models. It transforms prompts that work well with other LLMs into prompts that are optimized for LLM models, improving performance and reliability.

Key Benefits:

• No More Trial and Error: Stop manually tweaking prompts to get better results
• Fast Optimization: Get model-optimized prompts in minutes with template-based optimization
• Data-Driven Improvements: Use your own examples to create prompts that work for your specific use case
• Measurable Results: Evaluate prompt performance with customizable metrics

Requirements

To get started with prompt-ops, you'll need:

• Existing System Prompt: Your existing system prompt that you want to optimize
• Existing Query-Response Dataset: A JSON file containing query-response pairs (as few as 50 examples) for evaluation and optimization (see prepare your dataset below)
• Configuration File: A YAML configuration file (config.yaml) specifying model hyperparameters, and optimization details (see example configuration)

How It Works

┌──────────────────────────┐  ┌──────────────────────────┐  ┌────────────────────┐
│  Existing System Prompt  │  │  set(query, responses)   │  │ YAML Configuration │
└────────────┬─────────────┘  └─────────────┬────────────┘  └───────────┬────────┘
             │                              │                           │
             │                              │                           │
             ▼                              ▼                           ▼
         ┌────────────────────────────────────────────────────────────────────┐
         │                     prompt-ops migrate                       │
         └────────────────────────────────────────────────────────────────────┘
                                            │
                                            │
                                            ▼
                                ┌──────────────────────┐
                                │   Optimized Prompt   │
                                └──────────────────────┘

Simple Workflow

1. Start with your existing system prompt: Take your existing system prompt that works with other LLMs (see example prompt)
2. Prepare your dataset: Create a JSON file with query-response pairs for evaluation and optimization
3. Configure optimization: Set up a simple YAML file with your dataset and preferences (see example configuration)
4. Run optimization: Execute a single command to transform your prompt
5. Get results: Receive a model-optimized prompt with performance metrics

Real-world Results

HotpotQA

These results were measured on the HotpotQA multi-hop reasoning benchmark, which tests a model's ability to answer complex questions requiring information from multiple sources. Our optimized prompts showed substantial improvements over baseline prompts across different model sizes.

Quick Start (5 minutes)

Step 1: Installation

Note: We recommend installing from source as we are currently transitioning package names on PyPI. This ensures you get the latest stable version without any naming conflicts.

# Create a virtual environment
conda create -n prompt-ops python=3.10
conda activate prompt-ops

# Recommended: Install from source
git clone https://github.com/meta-llama/prompt-ops.git
cd prompt-ops
pip install -e .

# Alternative: Install from PyPI (may have naming transition issues, still on version 0.0.7)
# pip install llama-prompt-ops

Step 2: Create a sample project

This will create a directory called my-project with a sample configuration and dataset in the current folder.

prompt-ops create my-project
cd my-project

Step 3: Set Up Your API Key

Add your API key to the .env file:

OPENROUTER_API_KEY=your_key_here

prompt-ops uses LiteLLM as a unified API client. LiteLLM automatically detects the provider from your model name (e.g., openrouter/model, groq/model) and looks for the corresponding provider-specific environment variable (OPENROUTER_API_KEY, GROQ_API_KEY, etc.). For more inference provider options, see Inference Providers.

Step 4: Run Optimization

The optimization will take about 5 minutes.

prompt-ops migrate # defaults to config.yaml if --config not specified

Done! The optimized prompt will be saved to the results directory with performance metrics comparing the original and optimized versions.

To read more about this use case, we go into more detail in Basic Tutorial.

Prompt Transformation Example

Below is an example of a transformed system prompt from proprietary LM to Llama:

Preparing Your Data

To use prompt-ops for prompt optimization, you'll need to prepare a dataset with your prompts and expected responses. The standard format is a JSON file structured like this:

[
    {
        "question": "Your input query here",
        "answer": "Expected response here"
    },
    {
        "question": "Another input query",
        "answer": "Another expected response"
    }
]

If your data matches this format, you can use the built-in StandardJSONAdapter which will handle it automatically.

Custom Data Formats

If your data is formatted differently, and there isn't a built-in dataset adapter, you can create a custom dataset adapter by extending the DatasetAdapter class. See the Dataset Adapter Selection Guide for more details.

Multiple Inference Provider Support

prompt-ops supports various inference providers and endpoints to fit your infrastructure needs. See our detailed guide on inference providers for configuration examples with:

• OpenRouter (cloud-based API)
• vLLM (local deployment)
• NVIDIA NIMs (optimized containers)

Documentation and Examples

For more detailed information, check out these resources:

• Quick Start Guide: Get up and running with prompt-ops in 5 minutes
• Intermediate Configuration Guide: Learn how to configure datasets, metrics, and optimization strategies
• Dataset Adapter Selection Guide: Choose the right adapter for your dataset format
• Metric Selection Guide: Select appropriate evaluation metrics for your use case
• Inference Providers Guide: Configure different model providers and endpoints

Acknowledgements

This project leverages some of awesome open source projects including DSPy, thanks to the team for the inspiring work!

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.