PlayItOrNot

A personalized game prediction system that helps Steam users answer one practical question:

"If I buy this game, will I actually play it?"

PlayItOrNot combines multi-agent LLM reasoning, retrieval-augmented generation (RAG), and large-scale Steam gameplay data to deliver personalized predictions, explanations, and interactive demos via a Chrome Extension.

---

Table of Contents

• Overview
• Architecture
• Dataset
• Installation & Setup
• Usage
• System Components
• Evaluation
• Project Structure
• Examples
• License

---

Overview

Motivation

Steam users frequently purchase games they never play. This leads to wasted money, bloated libraries, and inaccurate recommendations. Traditional recommendation systems focus on "what you might enjoy", but not "will you actually play it?"

Key Features

• Multi-agent LLM pipeline (Extraction → Analyst → Critic)
• Per-feature user preference modeling using gameplay history
• Review-based RAG to ground reasoning in real player experiences
• Chrome extension demo integrated directly into Steam's UI
• Large-scale dataset: 786 users, 47k games, 2.6M reviews
• Fully open and reproducible pipeline

---

Architecture

System Diagram

Multi-Agent Pipeline Overview

1. Extraction Agent – Builds the user's preference profile using gameplay history and reviews
2. Analyst Agent – Predicts whether the user will actually play a candidate game
3. Critic Agent – Reviews and refines the Analyst's reasoning
4. RAG Engine – Provides feature-specific review evidence for grounded reasoning

---

Dataset

Source Users

We begin with the top game owners list from SteamDB:
https://steamdb.info/badge/13/

After removing private profiles, 786 valid public users remain.

Games

• Total collected: 47,753 games
• Invalid/unavailable games removed
• Flattened and stored in HuggingFace dataset

Reviews

For each game:

• English only
• 1-year window
• Sorted by helpfulness
• Off-topic removed
• Total: 2,631,430 reviews

Structured Data Formats

The pipeline converts raw Steam API responses into clean, normalized structured profiles for games, reviews, and users.

Full documentation of all structured fields, JSON schemas, and data objects is available in:
DATA_DETAILS.md

APIs Used

• Owned Games (Steam Web API)
  https://api.steampowered.com/IPlayerService/GetOwnedGames/v1/

• Game Metadata (Steam Store API)
  https://store.steampowered.com/api/appdetails

• Reviews (Steam Store API)
  https://store.steampowered.com/appreviews/{appid}

• SteamSpy Tags (SteamSpy API)
  https://steamspy.com/api.php?request=appdetails&appid={appid}

HuggingFace Dataset

https://huggingface.co/datasets/frankjc2022/steam-dataset

---

Installation & Setup

Requirements

• Python 3.10+
• OpenAI API key or compatible local model
• Steam API key

Environment Variables

Create a .env file:

STEAM_API_KEY=""
OPENAI_API_KEY=""

Steam API Key

A Steam Web API key is required to fetch a user's owned games and playtime.
You can generate one here: https://steamcommunity.com/dev/apikey

OpenAI API Key

An OpenAI API key is required if you use OpenAI models (e.g., gpt-4o, gpt-4o-mini) or OpenAI embedding models (e.g., text-embedding-3-small).
You can generate one here: https://platform.openai.com/api-keys

Installation

git clone https://github.com/ece1786-2025/PlayItAlot
cd PlayItAlot
pip install -r requirements.txt

---

Usage

Running the Pipeline

python scripts/run_all.py \
  --steamid 76561198017975643 \
  --appid 22380 \
  --model "gpt-4o" \
  --output-dir "results/76561198017975643_22380"

All Command Line Arguments for run_all.py

--steamid

Specify the user's SteamID. The user's profile must be public.

--appid

Specify the game's appid. This is the game you want to generate a prediction for.

--model

Specify the LLM model for agents. Default is gpt-4o-mini. OPENAI_API_KEY required in .env when using OpenAI API models. To see all supported models: python scripts/run_all.py --list-llm-models.

--base-url

Optional. Default is None. When using a different model backend (e.g., a local model such as openai/gpt-oss-20b), specify the base URL here.

--embedding-model

Specify the embedding model for review retrieval. This will override the embedding model defined in the config. To list all supported embedding models: python scripts/run_all.py --list-embedding-models.

--output-dir

Output directory where all prediction results will be saved.

--testing

Flag to run in testing mode. This removes the target appid from the user's profile to simulate the user not owning the game.

--redownload

Force re-download of Steam data, regardless of what is cached. This overwrites the corresponding setting in the config file.

--config

Path to the pipeline configuration file. Default: configs/pipeline.default.json.

Pipeline Configuration

The pipeline uses a JSON configuration file that defines default values such as embedding model, RAG settings, review limits, and caching behavior.

The default config is located at:
configs/pipeline.default.json

Some of these settings may be overridden using command-line arguments.

Listing Supported Models

python scripts/run_all.py --list-llm-models
python scripts/run_all.py --list-embedding-models

Chrome Extension Demo

1. Chrome → Extensions
2. Enable Developer Mode
3. Load unpacked → select chrome-extension/

The extension detects Steam's Add to Cart button and sends:

POST /api/predict

to the FastAPI server (server.py) and displays the prediction in a popup.

For the course demo, the server is hosted on a small AWS EC2 instance and kept running using PM2.

Demo Video

---

System Components

Extraction Agent

The Extraction Agent models the player's preferences using five gameplay facets grounded in psychological literature (King, Delfabbro & Griffiths, 2010):

• Social
• Manipulation & Control
• Narrative & Identity
• Reward & Punishment
• Presentation

It analyzes the user's top-played games and reviews to construct a structured user profile describing which facets matter most to them and why.

This profile forms the foundation for the Analyst Agent and for RAG retrieval.

---

Review Retrieval (RAG)

For each of the five gameplay facets defined during extraction, we retrieve semantically relevant reviews from a per-user vector database.

The feature-specific retrieval queries are defined in:
scripts/utils/feature_queries.py

The resulting top-k reviews become evidence blocks used by downstream LLM agents to support grounded reasoning.

---

Analyst Agent

The Analyst Agent evaluates a candidate game using:

• The user's extracted feature preferences
• Structured game metadata
• Feature-specific review evidence from RAG

Outputs include:

• Verdict: Likely, Unlikely, or Unsure
• Confidence score (0-1)
• A concise explanation referencing specific facets and evidence

---

Critic Agent

The Critic Agent validates the Analyst output:

• Checks for hallucinations or unsupported claims
• Ensures logical consistency
• Suggests corrections or refinements

This helps maintain reliability and coherence.

---

Evaluation

Human-Labeled Dataset

A manually labeled set of user-game pairs is stored in:
data/pattern_profiles_human.csv

Running Evaluation

python eval/eval_batch.py \
  --model gpt-4o-mini \
  --csv-path eval/dataset/pattern_profiles_human_test_set.csv

All Command Line Arguments for eval_batch.py

--model

Specify the LLM model for agents. Default is gpt-4o-mini. OPENAI_API_KEY required in .env when using OpenAI API models. To see all supported models: python scripts/run_all.py --list-llm-models.

--base-url

Optional. Default is None. When using a different model backend (e.g., a local model such as openai/gpt-oss-20b), specify the base URL here.

--multithread

Flag to enable multithreading for running the pipeline, improving evaluation speed.

--csv-path

Path to the CSV file used for evaluation. For example: eval/dataset/pattern_profiles_human_test_set.csv.

---

Project Structure

chrome-extension/                     # Chrome extension that hooks into Steam and displays predictions

server.py                              # FastAPI server used for the demo (receives steamid/appid and runs the pipeline)

configs/
└── pipeline.default.json              # Default configuration for model selection, settings, thresholds, etc.

notebooks/                             # Jupyter notebooks for analysis, EDA, prompt tuning, and evaluation
├── eda.ipynb
├── eval_analyst.ipynb
├── prompt_tuning_analyst.ipynb
└── prompt_tuning_analyst_reviews.ipynb

prompts/                               # All prompt templates explored for agents and ablations
├── analyst.system.txt
├── analyst.user.txt
├── critic.system.txt
├── critic.user.txt
├── extraction.system.txt
└── extraction.user.txt

scripts/
├── agents/                            # Multi-agent LLM logic
│   ├── extraction.py                  # Builds user feature profiles
│   ├── analyst.py                     # Predicts whether the user will play a new game
│   └── critic.py                      # Validates reasoning and improves reliability
│
├── utils/                             # Shared utility functions
│   ├── feature_queries.py             # RAG queries for each gameplay facet
│   ├── rag.py                         # SQLite-vec retrieval logic
│   ├── embeddings.py                  # Embedding model wrappers
│   ├── llm.py                         # LLM wrappers (OpenAI/local)
│   └── steam_api.py                   # Steam Web API wrapper
│
├── fetch_steam_data.py                # Collects raw user/game/review data
├── reviews_rag.py                     # Builds per-user vector DB for review retrieval
└── run_all.py                         # Main pipeline: fetch data → RAG → Extraction → Analyst → Critic

data/
├── raw/                               # Raw helper files (e.g., HTML user list used for parsing
├── cache/                             # Cached processed data to speed up repeated runs
│   ├── games/                         # Raw game metadata
│   ├── reviews/                       # Raw Steam reviews
│   ├── users/                         # Raw user-owned game lists
│   ├── structured_game_profile/       # Cleaned game features for LLM input
│   ├── structured_user_profile/       # Extracted user gameplay facet profiles
│   └── structured_user_reviews_profile/ # RAG review summaries per feature
│
└── rag/                               # Per-user SQLite vector DBs
    └── <steamid>_reviews_<model>.sqlite

eval/
├── dataset/                           # Human-labeled evaluation splits
├── runs/                              # Logs & outputs for batch experiments
└── eval_batch.py                      # Batch evaluation script

results/
└── <steamid_appid>/                   # Outputs from the full pipeline for each test pair

---

Examples

Below are representative artifacts from a single end-to-end user-game example.
Each user in the dataset has many games; we show only one here to illustrate the typical structure of prompts, agent outputs, and processed data inside the pipeline.

---

1. Agent Prompts & Outputs (Full Chain Example)

A complete run for one user–game pair is available here:

Example Run Folder

This folder contains the exact prompts and JSON outputs used by the Extraction, Analyst, and Critic agents.

Extraction Agent

• extraction.system.txt
• extraction.user.txt
• extraction_output.json

Analyst Agent

• analyst.system.txt
• analyst.user.txt
• analyst_output.json

Critic Agent

• critic.system.txt
• critic.user.txt
• critic_output_1.json
• critic_output_2.json

This collection shows the full reasoning chain: Extraction → Analyst → Critic.

---

2. Sample Data (Raw → Structured → RAG)

Representative input data and processed artifacts are available here:

Example Data Folder

Raw Steam Data

• game.json
• reviews.json
• steamspy.json
• user_raw.json

Combined Profiles

• user_full_profile.json
• user_full_profile_with_holdout.json

Structured Profiles (Normalized Steam Data)

• structured_game_profile.json
• structured_user_profile.json
• structured_user_reviews_profile.json

RAG Vector Database (Per-User)

• reviews.sqlite

These examples illustrate the entire data flow:
Raw Steam Data → Normalized Structured Profiles → RAG Vector DB → Multi-Agent Pipeline Outputs

---

License

This project is intended for educational and research purposes only.
All third-party APIs, data sources, and server components belong to their respective owners.