Awesome DSPy

A curated list of awesome examples and resources for DSPy.

Table of Contents

• Agents
• Dataloaders
• Functional
• Integrations
• Longform QA
• Math
• NLI
• QA
• Quiz
• Text to SQL
• Tweets
• Videos
• Papers
• Benchmarks

Agents

• multi_agent.ipynb: Multi-agent system examples.
• multi_agent_llama3.ipynb: Multi-agent system using Llama3.
• Building a chess playing agent using DSPy by Franck SN: Creating a chess-playing agent with DSPy.

Dataloaders

• dataloaders_dolly.ipynb: Dataloaders for the Dolly dataset.
• dolly_subset_100_rows.csv: Subset of the Dolly dataset with 100 rows.

Functional

• functional.ipynb: Functional programming in Python.
• repl.py: REPL for testing functional programming concepts.
• signature_opt_typed.ipynb: Optimized function signatures with type hints.
• Using DSPy to train Gpt 3.5 on HumanEval by Thomas Ahle: Training Gpt 3.5 on HumanEval.

Integrations

• clarifai_llm_retriever_example.ipynb: Example integration with Clarifai LLM retriever.
• readme.md: Documentation for integrations.
• Haize Lab's Red Teaming with DSPy and see their DSPy code
• Using Ollama with DSPy for Mistral (quantized) by @jrknox1977

Longform QA

• longformqa_assertions.ipynb: Assertions in long-form question answering.
• utils.py: Utility functions for long-form QA.
• Long-form Answer Generation with Citations, by Arnav Singhvi: Applying DSPy Assertions in long-form answer generation.

Math

• CoT.ipynb: Chain of Thought (CoT) in math problems.
• turbo_8_8_10_gsm8k_200_300.json: GSM8K dataset for Turbo model.

NLI

• scone-cot_fewshot-turbo-gpt4-demos.json: SCONE dataset demos with CoT and few-shot learning.
• scone.ipynb: SCONE dataset examples.
• Indian Languages NLI with gains due to compiling by Saiful Haq: Indian Languages NLI examples.
• Sophisticated Extreme Multi-Class Classification, IReRa, by Karel D’Oosterlinck

QA

• hotpotqa_with_assertions.ipynb: HotpotQA with assertions.
• hotpotqa_with_MIPRO.ipynb: HotpotQA with MIPRO.
• multihop_finetune.ipynb: Multi-hop finetuning for QA.
• DSPy on BIG-Bench Hard Example, by Chris Levy: BIG-Bench Hard Example with DSPy.

Quiz

• quiz_assertions.ipynb: Quiz assertions.
• Generating Answer Choices for Quiz Questions, by Arnav Singhvi: Applying DSPy Assertions in quiz question generation.

Text to SQL

• financial_data_text_to_sql.ipynb: Converting financial data to SQL queries.
• Optimizing Performance of Open Source LM for Text-to-SQL using DSPy and vLLM, by Juan Ovalle: Text-to-SQL optimization with DSPy.

Tweets

• compiling_langchain.ipynb: Compiling LangChain examples.
• tweet_metric.py: Metrics for analyzing tweets.
• tweets_assertions.ipynb: Assertions in tweet analysis.
• Generating Tweets for QA, by Arnav Singhvi: Applying DSPy Assertions in tweet generation.
• AI feedback, or writing LM-based metrics in DSPy: Writing LM-based metrics in DSPy.

Videos

• Getting Started with RAG in DSPy!
• DSPy Explained!
• Adding Depth to DSPy Programs
• Structured Outputs with DSPy

Papers

• [Oct'23] DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
• [Jan'24] In-Context Learning for Extreme Multi-Label Classification
• [Dec'23] DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
• [Dec'22] Demonstrate-Search-Predict: Composing Retrieval & Language Models for Knowledge-Intensive NLP
• Using DSPy, "The Unreasonable Effectiveness of Eccentric Automatic Prompts" (paper) by VMware's Rick Battle & Teja Gollapudi, and interview at TheRegister

Benchmarks

• DSPy Optimizers Benchmark on a bunch of different tasks, by Michael Ryan