Dataset Creation Scripts

Ready-to-run scripts for creating Hugging Face datasets from local files.

Available Scripts

📄 pdf-to-dataset.py

Convert directories of PDF files into Hugging Face datasets.

Features:

• 📁 Uploads PDFs as dataset objects for flexible processing
• 🏷️ Automatic labeling from folder structure
• 🚀 Zero configuration - just point at your PDFs
• 📤 Direct upload to Hugging Face Hub

Usage:

# Basic usage
uv run pdf-to-dataset.py /path/to/pdfs username/my-dataset

# Create private dataset
uv run pdf-to-dataset.py /path/to/pdfs username/my-dataset --private

# Organized by categories (folder structure creates labels)
# /pdfs/invoice/doc1.pdf → label: "invoice"
# /pdfs/receipt/doc2.pdf → label: "receipt"
uv run pdf-to-dataset.py /path/to/organized-pdfs username/categorized-docs

Output Format: The script creates a dataset where each example contains a pdf object that can be processed using the datasets library. Users can then extract text, convert to images, or perform other operations as needed.

from datasets import load_dataset

# Load your uploaded dataset
dataset = load_dataset("username/my-dataset")

# Access PDF objects
pdf = dataset["train"][0]["pdf"]

Requirements:

• Directory containing PDF files
• Hugging Face account (for uploading)
• No GPU needed - runs on CPU

Installation

No installation needed! Just run with uv:

# Run directly from GitHub
uv run https://huggingface.co/datasets/uv-scripts/dataset-creation/resolve/main/pdf-to-dataset.py --help

# Or clone and run locally
git clone https://huggingface.co/datasets/uv-scripts/dataset-creation
cd dataset-creation
uv run pdf-to-dataset.py /path/to/pdfs my-dataset

Authentication

Scripts use Hugging Face authentication:

1. Pass token via --hf-token argument
2. Set HF_TOKEN environment variable
3. Use cached credentials from huggingface-cli login

Examples

Create a Dataset from Research Papers

uv run pdf-to-dataset.py ~/Documents/papers username/research-papers

Organize Documents by Type

# Directory structure:
# documents/
# ├── invoices/
# │   ├── invoice1.pdf
# │   └── invoice2.pdf
# └── receipts/
#     ├── receipt1.pdf
#     └── receipt2.pdf

uv run pdf-to-dataset.py documents/ username/financial-docs
# Creates dataset with labels: "invoices" and "receipts"

Tips

• Large PDFs: The script handles large PDFs efficiently by uploading them as objects
• Organization: Use subdirectories to automatically create labeled datasets
• Privacy: Use --private flag for sensitive documents
• Processing: After upload, use the datasets library to extract text, images, or metadata as needed

License

MIT