tweaktune is a Rust-powered, Python-facing library for synthesizing datasets for training and fine-tuning AI models, especially Language Models.

Build powerful data pipelines to generate synthetic text, structured JSON, and function calling datasets using LLM APIs. Perfect for creating high-quality training data for model fine-tuning.

---

Documentation

• Complete Documentation - Comprehensive guides and API reference
• Examples - Working code examples
• Getting Started Guide - Quick start tutorial

---

Features

Flexible Data Sources

Load data from multiple sources:

• Files: Parquet, CSV, JSONL, JSON
• Databases: PostgreSQL, MySQL, SQLite (via ConnectorX)
• HuggingFace: Direct integration with datasets
• Arrow: PyArrow datasets and record batches
• Python: Dictionaries, functions, Pydantic models
• APIs: OpenAPI specifications for function calling
• SQL: Filter and transform with SQL queries

LLM Integration

Connect to any LLM provider:

• OpenAI: GPT-4, GPT-3.5, and compatible APIs
• Azure OpenAI: Enterprise deployments
• Local Models: Unsloth, MistralRS support
• Custom APIs: Any OpenAI-compatible endpoint

Powerful Pipeline Features

• Parallel Processing: Multi-worker execution for speed
• Dynamic Templates: Jinja2 templating with custom filters
• Data Validation: JSON schema, language detection, custom validators
• Deduplication: Exact hash, fuzzy simhash, semantic embeddings
• Quality Checks: Built-in and custom quality filters
• Conditional Logic: If-else branching in pipelines
• Custom Steps: Extend with Python classes
• Metadata Tracking: Track runs, items, and deduplication state

Dataset Generation

Create datasets for:

• Question-Answer pairs: Synthetic Q&A for training
• Function calling: Tool use and API interaction datasets
• Conversations: Multi-turn dialogue datasets
• Structured output: JSON conforming to schemas
• Chat formatting: Model-specific conversation formatting

---

Quick Start

Installation

pip install tweaktune

Simple Example

Generate synthetic data in minutes:

from tweaktune import Pipeline
import os

# Create a Q&A dataset
(Pipeline()
    .with_workers(3)
    .with_llm_openai(
        name="gpt4",
        api_key=os.environ["OPENAI_API_KEY"],
        model="gpt-4o-mini"
    )
    .with_template("system", "You are an expert educator.")
    .with_template("question", "Generate a question about: {{topic}}")
    .with_template("answer", "Answer this question: {{question}}")
    .with_template("output", """{"topic": "{{topic}}", "question": "{{question}}", "answer": "{{answer}}"}""")
    .iter_range(100)
        .add_column("topic", lambda data: f"Topic {data['index']}")
        .generate_text(
            template="question",
            llm="gpt4",
            output="question",
            system_template="system"
        )
        .generate_text(
            template="answer",
            llm="gpt4",
            output="answer",
            system_template="system"
        )
        .write_jsonl(path="qa_dataset.jsonl", template="output")
    .run())

Function Calling Dataset

Create datasets for training models on tool use:

from tweaktune import Pipeline
from pydantic import Field

def search_products(
    query: str = Field(..., description="Search query"),
    category: str = Field(..., description="Product category")
):
    """Search for products in the catalog."""
    pass

(Pipeline()
    .with_workers(5)
    .with_llm_openai("gpt4", api_key, "gpt-4o-mini")
    .with_tools_dataset("tools", [search_products])
    .iter_range(50)
        .sample_tools("tools", 1, "tool")
        # Generate user question, tool call, and response
        # ... (see examples/08_function_calling.py for complete code)
        .render_conversation(
            conversation="@user:question|@assistant:tool_calls([call])|@tool:result|@assistant:answer",
            tools="tool",
            output="conversation"
        )
        .write_jsonl(path="function_calling.jsonl", value="conversation")
    .run())

More examples in the examples directory.

---

Learn More

Documentation

• Getting Started - Installation and first pipeline
• Pipeline Basics - Understanding pipelines
• Data Sources - Loading data from various sources
• Templates - Using Jinja templates
• LLM Integration - Connecting to LLMs
• Pipeline Steps - Complete step reference
• Custom Steps - Creating custom steps
• Validation & Quality - Data validation and deduplication
• Conversation & Tools - Function calling datasets
• Chat Templates - Formatting for fine-tuning
• Metadata & Tracking - Pipeline metadata
• Advanced Features - Advanced patterns

Examples

• Simple Pipeline - Basic usage
• Data Sources - Loading various data formats
• Templates - Template examples
• Transformations - Data transformations
• Text Generation - LLM text generation
• JSON Generation - Structured output
• Q&A Dataset - Question-answer pairs
• Function Calling - Tool use datasets
• Conversations - Multi-turn dialogues
• Deduplication - Deduplication methods
• Validation - Data validation
• Custom Steps - Custom pipeline steps
• Conditional Logic - If-else branching
• Chat Templates - Chat formatting

---

Use Cases

• Model Fine-tuning: Generate training datasets for language models
• Function Calling: Create datasets for tool use and API interactions
• Conversation Data: Build multi-turn dialogue datasets
• Synthetic Data: Generate realistic data for testing and development
• Data Augmentation: Expand existing datasets with variations
• Quality Filtering: Clean and validate datasets with built-in checks
• RAG Datasets: Create question-answer pairs from documents

---

Key Features

Pipeline Steps

Chain together powerful steps:

• Data: sample(), read(), filter()
• Generation: generate_text(), generate_json(), generate_structured()
• Transformation: add_column(), mutate(), map(), ifelse()
• Validation: validate_json(), validate_tools(), check_language()
• Deduplication: check_hash(), check_simhash(), check_embedding()
• Rendering: render(), render_conversation(), render_tool_call()
• Output: write_jsonl(), write_csv(), print()
• Custom: .step() for your own Python classes

Why tweaktune?

• Fast: Rust-powered core for high performance
• Flexible: Python API for easy customization
• Scalable: Parallel processing with configurable workers
• Complete: End-to-end solution from data loading to output
• Type-safe: Pydantic integration for structured output
• Modern: Built for LLM fine-tuning workflows

---

Contributing

We welcome contributions! Feel free to open issues, suggest features, or create pull requests.

Please note that by contributing to this project, you agree to the terms of the Contributor License Agreement (CLA).

Development

# Clone the repository
git clone https://github.com/qooba/tweaktune.git
cd tweaktune

# Build the Python package (development mode)
make pyo3-develop

# Run tests
pytest

---

License

This project is licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT License (LICENSE-MIT)

at your option.

---

Links

• PyPI: https://pypi.org/project/tweaktune/
• GitHub: https://github.com/qooba/tweaktune
• Documentation: docs/
• Examples: examples/

---

Acknowledgments

Built with:

• Rust - Core performance
• PyO3 - Python bindings
• Polars - Fast DataFrames and SQL
• Minijinja - Templating (via minijinja)