---

Overview

This public GPLv3 repository provides a production-grade implementation of the Reinforcement Learning from Human Feedback (RLHF) pipeline together with its companion inference and systems modules. It mirrors the post-training infrastructure used by major research labs, optimized for consumer hardware, including CPU-only environments with zero GPU requirement.

The codebase includes implementations of 7 distinct preference optimization algorithms alongside process reward modeling, verifier-guided search, and advanced inference techniques such as Monte Carlo Tree Search (MCTS), A* decoding, Best-of-N reranking, speculative decoding, and hidden-deliberation serving contracts. It is designed for researchers and developers who want full control over post-training and test-time compute behavior without relying on commercial APIs or opaque hosted tooling.

As of March 2026, this pipeline has produced publicly released GGUF-quantized models trained entirely on consumer CPU hardware, demonstrating end-to-end viability from raw dataset to deployable artifact.

---

Mission

The exclusive control over post-training infrastructure has allowed a few organizations to artificially monopolize AI capabilities. They claim innovation while simply gating access to reinforcement learning, reward modeling, verifier-guided search, and test-time compute techniques. This repository is released under GPLv3 so the stack can be studied, modified, reproduced, and extended in the open.

This repository dismantles that barrier. By open-sourcing a complete RLHF runtime plus its surrounding inference, search, telemetry, and merge/export surfaces, the goal is to put reproduction of high-end post-training capability directly into the hands of the open-source community and reduce reliance on closed-source alignment and reasoning stacks.

---

Quick Start

# Clone and setup
git clone https://github.com/calisweetleaf/Reinforcement-Learning-Full-Pipeline.git
cd Full-RLHF-Pipeline

# Create environment
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt

# Verify installation
python -c "from rlhf import RLHFOrchestrator; print('Environment configured.')"

Training Example

# Train Qwen3 1.7B with SimPO (Memory efficient, reference-free)
python scripts/train_qwen3_1.7b.py --method simpo --epochs 2 --device cpu

# Train with DPO on GPU
python scripts/train_qwen3_1.7b.py --method dpo --epochs 3 --device cuda

---

RLHF Methods

Direct Optimization

Gradient-based preference learning without explicit reward modeling

Reinforcement Learning

Policy optimization against a reward signal

---

Features

Inference Logic

The repository includes an experimental inference and test-time-compute stack grounded in public work on process supervision, verifier-guided search, hidden deliberation, and reasoning-oriented reinforcement learning. The exposed runtime primitives include PRM-aware reranking, MCTS, A* search, hidden-deliberation serving, and search-oriented runtime helpers.

from inference_optimizations import MCTSGenerator, BestOfNSampler

# Best-of-N Sampling
sampler = BestOfNSampler(policy_model, reward_model)
result = sampler.generate(prompt, n_samples=16)

# Monte Carlo Tree Search
mcts = MCTSGenerator(policy_model, value_model, tokenizer)
result = mcts.generate(prompt, max_length=512)

---

Architecture

graph TB
    subgraph "Stage 1: Foundation"
        SFT[SFT Training]
        DATA[Dataset - maggiepie300k]
    end
    
    subgraph "Stage 2: Preference Learning"
        RM[Reward Model]
        DPO[DPO]
        GRPO[GRPO]
        SimPO[SimPO]
        KTO[KTO]
        PPO[PPO]
    end
    
    subgraph "Stage 3: Refinement"
        SP[Self-Play]
        IR[Iterative Refiner]
    end
    
    subgraph "Inference"
        MCTS[MCTS]
        BoN[Best-of-N]
        SD[Speculative]
    end

    subgraph "Export"
        MERGE[Model Merge - base + LoRA]
        GGUF[GGUF Quantization]
        Q4[Q4_K_M - default]
        Q5[Q5_K_M]
        Q8[Q8_0]
        F16[F16]
        Q3[Q3_K_M]
    end
    
    DATA --> SFT
    SFT --> RM
    SFT --> DPO
    SFT --> GRPO
    SFT --> SimPO
    SFT --> KTO
    RM --> PPO
    
    SFT --> MERGE
    MERGE --> GGUF
    GGUF --> Q4
    GGUF --> Q5
    GGUF --> Q8
    GGUF --> F16
    GGUF --> Q3
    
    DPO --> SP
    GRPO --> SP
    SimPO --> SP
    KTO --> SP
    PPO --> SP
    
SP --> IR
IR --> MCTS
IR --> BoN
IR --> SD

---

Cherry Technical Documentation

Cherry's advanced training, search, process-supervision, and inference work is documented here:

• Cherry SOTA Inference Upgrade — August 6, 2026
• Cherry Process Supervision & Moonshine Canon — August 6, 2026

Search-to-Learning Path

Cherry connects inference-time search back to policy optimization rather than treating search as a disconnected decoding utility.

MCTS and A* operate over branch-structured reasoning trajectories using verifier, reward, value, and process-reward signals. TreeRolloutCollector preserves search topology and sibling relationships when converting successful and unsuccessful branches into grouped rollout samples. Tree-GRPO then derives relative advantages across those branches and applies policy updates over the search-derived trajectories.

This gives Cherry two complementary operating modes:

1. Test-time search — additional compute can be allocated to MCTS, A*, Best-of-N, verifier-guided evaluation, PRM scoring, speculative decoding, and process-aware pruning.
2. Search-informed training — branch outcomes discovered by search can become structured policy-optimization signal through Tree-GRPO rather than disappearing after inference.

The August 2026 inference implementation also adds radix-prefix lookup, shared physical KV pages, branch-aware sequence forking, and copy-on-write handling of shared partial pages so tree-search descendants can reuse common prefix state rather than redundantly materializing identical ancestry.

---

Repo Surfaces

These files are part of the current public repo surface and are kept in-repo because they document the runtime and implementation context around the core training stack.

The public landing page prioritizes the canonical runtime docs and direct literature lineage over internal synthesis notes.

---

Research Lineage

This experimental inference/search update is positioned as a synthesis of public literature and implementation work, not as a claim around any proprietary system label. The most relevant paper line for the current public runtime surface includes:

• Let's Verify Step by Step
• Let's Reinforce Step by Step
• Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
• Fast Quiet-STaR: Thinking Without Thought Tokens
• AlphaZero-Like Tree-Search can Guide Large Language Model Decoding and Training
• StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Research Credits

The process-supervision, search, and reinforcement-learning lineage represented here builds on work by researchers including:

• Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe — Let's Verify Step by Step.
• Sarah Pan, Vladislav Lialin, Sherin Muckatira, and Anna Rumshisky — Let's Reinforce Step by Step.
• Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang — AlphaZero-Like Tree-Search can Guide Large Language Model Decoding and Training.
• Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman — Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking.
• Forest Agostinelli and collaborators — Q* Search: Heuristic Search with Deep Q-Networks.
• DeepSeek-AI — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.

These works form part of the public research lineage surrounding process supervision, tree-search-guided reasoning, reinforcement learning over reasoning trajectories, and test-time computation explored by Cherry.

---

Method Selection Guide

---

Validated Training Run

On March 7, 2026, this pipeline completed its first full end-to-end training and release run. All training was performed on CPU only — no GPU hardware was used at any stage.

This run validates the core claim of this repository: a complete post-training pipeline is reproducible on consumer hardware without GPU access. The SFT adapter was trained against the maggiepie300k dataset using the scripts/train_qwen3_1.7b.py entry point and merged with the Qwen3 1.7B base weights using model_merging.py's deterministic SHA256-manifest artifact save. The merged model was then quantized to GGUF format across multiple bit-depths for broad deployment compatibility.

---

GGUF Releases

Five GGUF quantizations of the merged Qwen3 1.7B SFT model are available. These represent the post-SFT, post-merge state of the model — before any DPO or further preference optimization stage.

The Q4_K_M format is the recommended default for most users. It provides the best trade-off between inference speed, memory usage, and output quality at the 1.7B parameter scale.

# Load a GGUF quant with llama.cpp or any compatible runtime
./llama-cli -m qwen3-1.7b-sft-merged.Q4_K_M.gguf -p "Your prompt here" -n 256

---

Model Merging

The model_merging.py module provides a full suite of stateless, deterministic merge algorithms. All methods operate in float32 internally and cast back to the original parameter dtype to prevent silent precision loss with bfloat16/float16 checkpoints.

from model_merging import ModelMerger, MergeConfig, save_merge_artifact

config = MergeConfig(method="task_arithmetic", density=0.8, scaling_coef=1.0)
merger = ModelMerger(config)

merged = merger.merge(base_model, [sft_adapter_model])

# Deterministic SHA256 manifest — provenance tracking for every artifact
save_merge_artifact(
    merged,
    output_dir="./merged_output",
    metadata={"run_date": "2026-03-07", "dataset": "maggiepie300k"},
    merge_config=config,
)

---

Benchmark Harness

The benchmark_harness.py module wraps the full inference and merge stacks in a standardized evaluation matrix, a budget-aware profile selector, and a regression gate for CI workflows.

from benchmark_harness import BenchmarkHarness, BudgetProfileSelector, RegressionGate

# Profile selection based on hardware constraints
selector = BudgetProfileSelector()
merge_cfg, bon_cfg, mcts_cfg, spec_cfg = selector.select({
    "max_ram_gb": 8,
    "has_gpu": False,
    "latency_budget_ms": 500,
    "quality_priority": True,
})

# Run merge benchmark across all strategies
harness = BenchmarkHarness()
results = harness.run_merge_benchmark(base, [ft_model], strategies=["ties", "dare", "task_arithmetic"])
ranked = harness.rank_strategies(results)
harness.emit_report(results, "bench.json")

# Regression gate for CI promotion
gate = RegressionGate(thresholds={"quality_floor": 0.7, "latency_ceiling_ms": 800, "ram_ceiling_gb": 12})
ok = gate.check(current_results, baseline_results)

Available hardware profiles:

---

Implementation Notes

• No Safety Filtering: This pipeline applies no inherent safety filtering or aligned rejection sampling. The model's behavior is determined solely by the data provided.
• Modular Design: All components (Trainers, models, strategies) are decoupled to allow for custom implementations.
• Production Ready: Code is structured for maintainability and scalability, handling logging, checkpointing, and error recovery robustly.
• CPU-Only Viable: The SFT training loop, LoRA adapter training, model merging, and telemetry collection are all validated on CPU hardware. GPU is supported and recommended for larger models and longer runs, but is not required.
• Inference Protocol Layer: inference_protocols.py provides PolicyAdapter, RewardScorerAdapter, and ValueScorerAdapter wrappers that satisfy typed Protocols without assuming specific model attributes. All inference components consume these adapters rather than raw model objects, enabling drop-in swaps of any compatible model.
• Process-Supervision And Search: The companion inference lane includes process-reward-aware reranking, verifier-guided search utilities, lexical-MDP abstractions, and hidden-deliberation answer-serving primitives for experimental test-time compute workflows.
• Telemetry: telemetry.py provides a TelemetryRecorder singleton with thread-safe reservoir sampling (Vitter's Algorithm R) for latency quantiles, speculative acceptance rates, KV-cache hit ratios, and MCTS expansion counts. Snapshots are emittable as JSON at any point in a run.
• Deterministic Artifact Provenance: All merge artifacts include a SHA256 manifest of input and output state dicts, recorded via save_merge_artifact(). This enables exact reproduction and audit of any released checkpoint.

---

Provenance and Citation

If you use any methods, code, or repository in your research, please cite it and follow the current GPLv3 distribution terms used by this public repo.

@misc{https://doi.org/10.5281/zenodo.18607464, doi = {10.5281/ZENODO.18607464}, url = {https://zenodo.org/doi/10.5281/zenodo.18607464}, author = {Rowell, Christian Trey Levi}, keywords = {RLHF, DPO, PPO, Reinforcement Learning}, title = {Reinforcement Learning Multi Method Pipeline and Inference tooling}, publisher = {Zenodo}, year = {2026}, copyright = {GNU General Public License v3.0}}

--

References

Process Supervision Let's Verify Step by Step. (2023).

Let's Reinforce Step by Step. (2023).

Hidden Deliberation Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking. (2024).

Fast Quiet-STaR: Thinking Without Thought Tokens. (2025).

Search And Reasoning AlphaZero-Like Tree-Search can Guide Large Language Model Decoding and Training. (2023).

StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization. (2025).

A Training Data Recipe to Accelerate A* Search with Large Language Models. (2024).

Direct Preference Optimization Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

SimPO Meng, Y., et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward.

DeepSeek-R1 DeepSeek-AI. (2024). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.

Proximal Policy Optimization Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms.

FlashAttention-2 Dao, T. (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning.