CS4: Evaluating LLM Creativity in Story Generation

Abstract

Evaluating the creativity of large language models (LLMs) in story writing is challenging since generated stories may resemble existing narratives in the models' training data. To address this, we introduce CS4 (Comparing the Skill of Creating Stories by Controlling the Synthesized Constraint Specificity), a benchmark dataset with prompts of varying specificity. By increasing prompt constraints, we prevent models from reproducing known stories, indirectly assessing their creativity.

Our experiments on models like LLaMA, Gemma, and Mistral show the difficulty LLMs face in balancing constraint satisfaction and narrative coherence, especially with highly specific prompts. We also demonstrate that Learning from Human Feedback (LHF), tested with OLMo, improves story selection but has limited impact on generating genuinely creative stories.

The datset can be found here.

Table of Contents

• Project Overview
• Installation
• Project Structure
• Usage
• Datasets
• Evaluation Scripts
• Results
• Citation
• License

Project Overview

This repository contains the code and data associated with the CS4 benchmark, designed to evaluate the creativity of large language models (LLMs) under various levels of constraint specificity. The benchmark allows us to investigate how LLMs balance creativity, constraint satisfaction, and coherence in story generation tasks. Additionally, the code includes evaluation scripts for the analysis of multiple LLMs' performance on CS4.

Key Models Evaluated:

• LLaMA
• Gemma
• Mistral
• OLMo (with insights into the impact of Learning from Human Feedback)

Key Components:

1. CS4 Dataset: Contains prompts with varying degrees of specificity to test the creativity of LLMs.
2. Evaluation Scripts: To measure constraint satisfaction, coherence, and perplexity of the generated stories.

Installation

1. Clone the repository:

   git clone https://github.com/anirudhlakkaraju/cs4_benchmark.git
   cd cs4_benchmark
   

2. Set up the environment:

   Choose one of the following methods:

   Option A: Using pip (Recommended)

   python -m venv myenv
   source myenv/bin/activate  # On Windows, use: myenv\Scripts\activate
   

   Option B: Using Conda

   If you don't have Miniconda, download it from Miniconda's website.

   conda create --name myenv python=3.11.7
   conda activate myenv
   

3. Install dependencies:

   pip3 install -r requirements.txt
   

4. Set up API keys:

   Choose one of the following methods:

   Option 1: Using a .env file (Recommended)

   1. Create a .env file in your project directory.
   2. Add your API key:

      OPENAI_API_KEY=your_openai_api_key
      

   3. Load the variables in your code:

      from dotenv import load_dotenv
      load_dotenv()
      

   Option 2: Exporting directly in the terminal

   • For Linux/Mac:

     export OPENAI_API_KEY=your_openai_api_key
     

   • For Windows (Command Prompt):

     set OPENAI_API_KEY=your_openai_api_key
     

5. Access the key in your code:

   import os
   openai_api_key = os.getenv('OPENAI_API_KEY')
   

Now you're ready to use the CS4 benchmark and evaluation scripts!

Project Structure

The project is organized as follows:

cs4_benchmark/
│
├── input_files/           # Directory for input data
│
├── output_files/          # Directory for generated outputs (created during evaluation)
│
├── myenv/                 # Virtual environment (not tracked in Git)
│
├── eval_execution.log     # Log file for evaluation execution (created during evaluation)
│
└── [other project files and directories]

• input_files/: Contains the input data for the models and evaluations.
• output_files/: Stores the results and outputs generated by the evaluation scripts. This directory is created automatically when you run the evaluation.
• myenv/: The virtual environment directory (you should not commit this to version control).
• eval_execution.log: Log file that captures the execution details of evaluation runs. This file is generated when you run the evaluation.

Note: The output_files/ directory and eval_execution.log file are created dynamically when you execute the evaluation scripts. They will not exist in the project structure until after the first evaluation run.

Ensure that you have the necessary permissions to create and write to the output_files/ directory and the eval_execution.log file. The evaluation scripts will read input data from input_files/ and save results to output_files/.

Usage

Running the Evaluation

You can run all evaluation scripts in one step using the run_evaluations.sh script. This script allows you to customize input and output paths easily.

1. Open run_evaluations.sh and modify the input paths, output directories, and file names as needed. Key parameters include:

   • Model Paths:

     MODEL1_PATH="input_files/D3_Gemma.csv"  # Path to Gemma model's output
     MODEL2_PATH="input_files/D3_Llama.csv"  # Path to Llama model's output
     MODEL3_PATH="input_files/D3_Mistral.csv"  # Path to Mistral model's output
     

     These paths point to the CSV files containing the outputs from each model (Gemma, Llama, and Mistral) for the CS4 benchmark tasks.

   • Model Labels:

     LABEL1="Gemma"  # Label for the Gemma model
     LABEL2="Llama"  # Label for the Llama model
     LABEL3="Mistral"  # Label for the Mistral model
     

     These labels are used in graphs and output files to identify each model.

   • Evaluation-specific Input Paths:

     INPUT_CONS_SATISF="input_files/D3_Gemma.csv"  # Input for constraint satisfaction evaluation
     INPUT_DIVERSITY_CALC="input_files/D3_Gemma.csv"  # Input for diversity calculation
     INPUT_COH_VS_CONS="input_files/D3_Gemma.csv"  # Input for coherence vs. consistency evaluation
     INPUT_JSON_QUC_AND_RCS="path/to/your/input.json"  # Input JSON for QUC and RCS evaluation
     
     

     These paths specify input files for different evaluations. They can be the same as one of the model paths or different files specific to each evaluation.

2. Make the script executable:

   chmod +x run_evaluations.sh
   

3. Run the script:

   ./run_evaluations.sh
   

The script will create the necessary output directories and run the evaluation with the specified parameters. Results will be saved in the output_files directory, organized into subdirectories for each type of evaluation (e.g., constraint satisfaction, diversity calculation, etc.).

For detailed information on the format of input files and the structure of output results, refer to the Datasets and Evaluation Scripts sections of this README.

Individual Scripts

Each evaluation script can also be run independently by providing the necessary arguments. For example, to evaluate constraint satisfaction:

python constraint_satisfaction.py \
    --input_path /path/to/input.csv \
    --output_path /path/to/output.csv

Datasets

CS4 Benchmark Dataset

The CS4 dataset is designed to evaluate LLM creativity by introducing prompts with varying levels of specificity:

• Low-Specificity Prompts: Fewer constraints, allowing for more creative freedom.
• High-Specificity Prompts: Many constraints, forcing the model to produce more structured outputs.

The dataset includes multiple stories generated by LLaMA, Gemma, Mistral, and OLMo models, all with different levels of instruction complexity.

Evaluation Scripts

Key Metrics:

1. Constraint Satisfaction: Measures how well the generated story adheres to the prompt's constraints.
2. Coherence: Evaluates the overall coherence of the generated narrative.
3. Diversity: Calculates n-gram diversity to assess story originality.
4. Perplexity: Measures the predictability and fluency of the generated text.
5. QUC and RCS: Metrics specifically designed for this benchmark to evaluate creativity under constraints.

The key scripts for evaluation include:

• coherence_vs_constraint_graph.py: Generates graphs comparing coherence vs. constraint satisfaction.
• constraint_satisfaction.py: Evaluates how well generated stories satisfy constraints.
• diversity_calculation.py: Calculates diversity in generated stories.
• perplexity_graph_generation.py: Plots perplexity against the number of constraints.
• quc_and_rcs.py: Computes and plots QUC and RCS scores.

Results

In our experiments:

• LLaMA, Gemma, and Mistral show distinct performance across varying levels of constraint specificity.
• Models struggle with maintaining creativity when the prompt becomes highly specific.
• OLMo demonstrates improved story selection via Learning from Human Feedback (LHF), but it struggles to generate unseen creative stories, even with LHF.

Citation

  title = {CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints},
  author = {Atmakuru*, Anirudh and Nainani*, Jatin and Bheemreddy*, Rohith Siddhartha Reddy and Lakkaraju*, Anirudh and Yao, Zonghai and Zamani, Hamed and Chang*, Haw-Shiuan},
  booktitle = {6th Workshop on Narrative Understanding (WNU)},
  year = {2024}
}

License

This project is licensed under the MIT License. See the LICENSE file for more details.

---

For more information on the dataset and experiments, please refer to the accompanying research paper.