SPaRC Solver

SPaRC: A Spatial Pathfinding and Reasoning Challenge

Commands

python3 sparc_puzzle_solver.py --dataset-dir datasets --batch-size X --max-puzzles X
python3 sparc_puzzle_solver.py --dataset datasets/puzzle_all_test.jsonl --batch-size X --max-puzzles X

A Python tool that uses Large Language Models (LLMs) to solve puzzles from the SPaRC dataset.

Overview

This program loads puzzles from a SPaRC dataset and uses an LLM (either local via vllm or via OpenAI API) to solve them. SPaRC contains grid puzzles that challenge spatial reasoning abilities.

Features

• Support for multiple puzzle types (dots, squares, shapes, stars, triangles, etc.)
• Rich terminal UI with colorful visualization
• Step-by-step solution explanations
• Flexibility to use either local models with vllm or API-based models
• Custom API endpoint support for OpenAI-compatible APIs

Project Structure

After refactoring, the project has the following structure:

.
├── README.md                # This file
├── config.yaml              # Configuration file
├── config.template.yaml     # Template for configuration
├── sparc_puzzle_solver.py   # Main entry point
├── prompt_generator.py      # Generates LLM prompts for puzzles
├── llm/                     # LLM providers
│   ├── __init__.py
│   ├── api_provider.py      # OpenAI/compatible API provider
│   ├── base_provider.py     # Base class for LLM providers
│   └── local_provider.py    # Local VLLM provider
├── solution/                # Solution processing
│   ├── __init__.py
│   ├── parser.py            # Extracts solution paths from LLM responses
│   └── validator.py         # Validates solution paths against known solutions
├── utils/                   # Utility modules
│   ├── __init__.py
│   ├── dataset_loader.py    # Dataset loading and filtering
│   └── stats.py             # Statistics collection and reporting
└── visualization/           # Visualization components
    ├── __init__.py
    ├── grid_formatter.py    # Formats grids for display
    └── solution_display.py  # Displays solutions with rich formatting

Installation

1. Clone this repository:

git clone https://github.com/yourusername/sparc-puzzle-solver.git
cd sparc-puzzle-solver

2. Install the required dependencies:

pip install -r requirements.txt

Configuration

Edit the config.yaml file to configure the LLM settings:

# LLM settings
llm_mode: "api"  # Options: "local" or "api"

# Local VLLM settings (when llm_mode is "local")
local_model_path: "TheBloke/Llama-2-13B-chat-GPTQ"  # Path to local model or Hugging Face model ID

# API settings (when llm_mode is "api")
api_base: "https://api.openai.com/v1"  # Default OpenAI API endpoint
api_model: "gpt-4"  # Model to use for API requests

Usage

Basic Usage

python sparc_puzzle_solver.py --dataset datasets/puzzle_database_all.jsonl

Solving a Specific Puzzle

python sparc_puzzle_solver.py --dataset datasets/puzzle_database_all.jsonl --puzzle-id "68829d17b557f9c0"

Limiting the Number of Puzzles

python sparc_puzzle_solver.py --dataset datasets/puzzle_database_all.jsonl --max-puzzles 5

Filtering by Difficulty Level

python sparc_puzzle_solver.py --dataset datasets/puzzle_database_all.jsonl --difficulty 3

Using Local Model

python sparc_puzzle_solver.py --dataset datasets/puzzle_database_all.jsonl --mode local

Using Custom API Endpoint

1. Configure your API endpoint in config.yaml
2. Run the program:

python sparc_puzzle_solver.py --dataset datasets/puzzle_database_all.jsonl

Dataset Format

The dataset should be a JSONL file (JSON Lines format) containing one puzzle object per line. Each puzzle has the following structure:

{
  "difficulty_level": 4,
  "difficulty_score": 4.61,
  "grid_size": {"height": 3, "width": 2},
  "id": "68829d17b557f9c0",
  "polyshapes": {"1584": "·█··\n·██·\n··█·\n····"},
  "puzzle_array": [["+","+","+","+","+"],["+","N","+","o-R","+"],["+","+","+","+","+"],["+","N","G","P-W-1584","+"],[".","+","+","+","+"],["+","o-P","+","N","+"],["+","S","+","+","E"]],
  "solution_count": 1,
  "solutions": [{"index": 0, "path": [{"x":1,"y":6}, {"x":2,"y":6}, ...]}],
  "text_visualization": "puzzle:\n  dimensions:\n    width: 2\n    height: 3\n  start:\n    x: 1\n    y: 6\n  end:\n    x: 4\n    y: 6\n  cells:\n..."
}

The puzzle format includes:

• difficulty_level: Numeric difficulty level (1-10)
• difficulty_score: Precise difficulty score
• grid_size: Width and height of the puzzle grid
• id: Unique puzzle identifier
• polyshapes: Definitions of Tetris-like shapes used in the puzzle
• puzzle_array: 2D array representing the puzzle grid with special notation
• solution_count: Number of valid solutions
• solutions: Array of valid solution paths
• text_visualization: Text-based description of the puzzle

Supported Puzzle Types

• dots: Puzzles with dots that must be crossed
• squares: Puzzles with colored squares that must be segregated
• straight_shapes: Puzzles with Tetris-like shapes
• slanted_shapes: Puzzles with shapes that can be rotated
• outlined_shapes: Puzzles with shapes that must be removed
• stars: Puzzles with stars that must be paired
• correctors: Puzzles with correctors that fix mistakes
• triangles: Puzzles with triangles indicating how many sides to fill
• mixed: Puzzles combining multiple elements

Links

• Website: sparc.gipplab.org
• Dataset: HuggingFace/lkaesberg/SPaRC

Requirements

Create a requirements.txt file with the following dependencies:

rich==13.5.2
pyyaml==6.0
openai==1.3.0
vllm==0.2.0

License

• Code: BSD 3-Clause
• Dataset: CC-BY-4.0

Acknowledgements

• The puzzle rules are based on the Steam guide by vernanonix
• The Witness is a game developed by Thekla, Inc.
• The SPaRC website is a fork of jbzdarkid's repository