RLCF - Reinforcement Learning from Computer-based Evaluator Feedback

This research investigates reinforcement learning-based fine-tuning of MarioGPT to enhance alignment with specific game level design preferences. A computer-based evaluator function was employed to score levels across playability, novelty, and aesthetic aspects. The dataset, generated by sampling over 9,900 levels from the original MarioGPT model and scoring them using the evaluator function, was used to train a reward model. This reward model assesses level designs and provide reward (i.e. the 3 scores), which were then utilized to fine-tune the original MarioGPT model through Proximal Policy Optimization (PPO).

How does it work?

This diagram illustrates the training process used to fine-tune MarioGPT, aligning it with the evaluator function's preferences to generate levels that are more playable, novel, and free from aesthetic defects.

User guide

• sampling_mariogpt.py Performs sampling on the MarioGPT model, facilitating the generation of level designs.
• playability_measure.py Evaluates and scores the playability of the Mario level designs, aiding in quality assessment.
• shannon_entropy.py Evaluates and scores the novelty of the Mario level designs, aiding in quality assessment.
• aesthetic_evaluator.py Evaluates and scores the aesthetic of the Mario level designs, aiding in quality assessment.
• preference_model.py The Reward/Preference model, which can be trained using the dataset generated by sampling along with evaluator function scores for the sampled levels.
• value_head.py Implementation of Value Head, which is used as value function during the PPO training of MarioGPT.
• ppo_trainer.py Implementation of the Proximal Policy Optimization (PPO) trainer to fine-tune the MarioGPT model by leveraging the preference model for optimized level design generation.

Currently, each step in the fine-tuning process must be executed sequentially. In the future, I would streamline this workflow into a single, unified pipeline for the ease of use.

Instruction to install

python setup.py install

How to run

You can refer the mariogpt guide to generate mario game level.

This process involves enhancing MarioGPT's game-level generation by fine-tuning it with Reinforcement Learning (RL) using computer-generated feedback.

• Step 1: Generate Game Levels

  • Sampling the Pre-trained Model: Use the pre-trained MarioGPT model sampling_mariogpt.py to generate various game-level designs based on diverse prompts. This step establishes a dataset of levels for evaluation.

• Step 2: Evaluate Generated Levels

  • Scoring Levels: Evaluate each generated level using the following evaluator functions:
  • Playability: Run playability_measure.py to assess how playable each level is.
  • Novelty: Use shannon_entropy.py to measure the novelty of each level.
  • Aesthetic Quality: Execute aesthetic_evaluator.py to score the visual and structural appeal.
  • Data Compilation: After scoring, all level data and corresponding scores will be saved into a CSV file, which will be used to train the reward model.

• Step 3: Train the Reward/Preference Model

  • Model Training: Run preference_model.py with the generated CSV file as input. This script trains a reward model to predict the "preferability" score of game levels, aligning with the evaluator's assessments. Ensure that the training parameters (e.g., learning rate, batch size) are set optimally.

• Step 4: Fine-Tune MarioGPT Using RL

  • Reinforcement Learning with PPO: Execute ppo_trainer.py with the reward model trained in the previous step. This script uses Proximal Policy Optimization (PPO) to fine-tune MarioGPT, guiding it to generate levels that align closely with the evaluator's feedback on playability, novelty, and aesthetics.

Validation and Testing: After fine-tuning, validate the updated MarioGPT model by generating new levels and evaluating them against the same metrics. Compare results to assess improvements in playability and novelty.

Requirements

• Python 3.12

Citation

@misc{https://github.com/tsmbalu/MarioGPT_RL,

author = {Balasubramani Murugan},

keywords = {Artificial Intelligence (cs.AI), Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},

title = {Enhancing Game Level Generation via RLCF-guided MarioGPT},

year = {2024}, }

Reference

@misc{https://doi.org/10.48550/arxiv.2302.05981, doi = {10.48550/ARXIV.2302.05981},

url = {https://arxiv.org/abs/2302.05981},

author = {Sudhakaran, Shyam and González-Duque, Miguel and Glanois, Claire and Freiberger, Matthias and Najarro, Elias and Risi, Sebastian},

keywords = {Artificial Intelligence (cs.AI), Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},

title = {MarioGPT: Open-Ended Text2Level Generation through Large Language Models},

publisher = {arXiv},

year = {2023},

copyright = {arXiv.org perpetual, non-exclusive license} }

Github: https://github.com/shyamsn97/mario-gpt