MarceLLo

Trained with poems and posts: an RL-based style transfer system that fine-tunes an LLM to capture my writing style. Research ongoing.

Concept

Standard fine-tuning (SFT) memorizes examples. MarceLLo uses GRPO (Group Relative Policy Optimization) to let the model discover writing style through reinforcement learning, guided by a style classifier as reward signal.

Same technique DeepSeek used for R1, but the reward is "how much does this sound like Marcelo" instead of "is this reasoning correct."

Architecture

graph TD
    subgraph "Phase 1: Data"
        A[Collect Writing Samples] --> B[Process & Clean]
        B --> C[Negative Sampling<br/>Contrastive Pairs]
    end

    subgraph "Phase 2: Reward Model"
        D[DeBERTa-v3-small<br/>+ Classification Head]
        D -->|"P(Marcelo) → 0..1"| E[Style Score]
    end

    subgraph "Phase 3: GRPO Training"
        F[Base Model: Qwen2.5-1.5B]
        F -->|"1. Generate G completions"| G[Group Sampling]
        G -->|"2. Score each"| E
        E -->|"3. A_i = r_i - mean / std"| H[Group-Relative Advantages]
        H -->|"4. Clipped policy gradient + KL"| F
    end

    subgraph "Phase 4: Evaluation"
        I[Style Score · Perplexity · Distinct-N<br/>A/B Comparison · Human Eval]
    end

    C --> D
    F --> I

GRPO Key Insight

GRPO eliminates the need for a separate value/critic model:

1. Generates a group of G outputs for the same prompt
2. Scores all G outputs with the reward model
3. Uses the group mean as baseline (no learned value function)
4. Computes advantages: A_i = (r_i - mean(r)) / std(r)
5. Updates policy with clipped surrogate objective + KL penalty

Project Structure

marcello/
├── configs/
│   ├── classifier.yaml    # Style classifier hyperparams
│   ├── grpo.yaml           # GRPO training config
│   └── data.yaml           # Data pipeline config
├── src/marcello/
│   ├── data/               # Collection, processing, negative sampling
│   ├── classifier/         # Style classifier (reward model)
│   ├── grpo/               # GRPO trainer, reward wrapper, sampling
│   ├── eval/               # Metrics, comparison, reporting
│   └── utils/              # Logging, helpers
├── scripts/                # Entry points for each phase
├── tests/                  # Unit tests
└── data/                   # Raw and processed datasets

Quick Start

# Install
pip install -e ".[dev]"

# Place writing samples in data/raw/writing_samples/ (.txt or .jsonl)

# Process data and generate contrastive pairs
python scripts/collect_data.py --config configs/data.yaml

# Train the style classifier (reward model)
python scripts/train_classifier.py --config configs/classifier.yaml

# Run GRPO training
python scripts/train_grpo.py --config configs/grpo.yaml

# Generate with the GRPO adapter
python scripts/generate.py --model outputs/grpo/final --prompt "La ciudad no duerme cuando siente miedo." --format-prompts

# Evaluate
python scripts/evaluate.py --model outputs/grpo/final --prompts data/eval_prompts.txt --format-prompts --output outputs/eval/latest.json

Models

Pre-trained Weights

To publish your own trained models after running the pipeline:

# Login once
huggingface-cli login

# Push everything (classifier + model + dataset)
python scripts/push_to_hub.py --all

# Or push individual artifacts
python scripts/push_to_hub.py --classifier
python scripts/push_to_hub.py --model
python scripts/push_to_hub.py --dataset

# Preview what would be pushed (no upload)
python scripts/push_to_hub.py --all --dry-run

# Push to your own org/user instead of the default
python scripts/push_to_hub.py --all --org your-hf-username

# Merge LoRA weights into the base model before pushing (standalone checkpoint)
python scripts/push_to_hub.py --model --merge-weights

To use the pre-trained models directly:

from transformers import AutoModelForSequenceClassification, AutoModelForCausalLM, AutoTokenizer

# Style classifier
classifier = AutoModelForSequenceClassification.from_pretrained(
    "marcelo-earth/marcello-style-classifier"
)

# Fine-tuned LLM
model = AutoModelForCausalLM.from_pretrained("marcelo-earth/marcello-qwen2.5-1.5b-grpo")
tokenizer = AutoTokenizer.from_pretrained("marcelo-earth/marcello-qwen2.5-1.5b-grpo")

Inference

The GRPO model is trained with explicit style/language control tags. For best results, wrap raw prompts with the same template used during training:

python scripts/generate.py \
  --model outputs/grpo/final \
  --prompt "The night felt larger than the street below." \
  --format-prompts \
  --style standard

If your prompt file already contains control tags, omit --format-prompts.

Resources

• GRPO Paper (DeepSeek-Math)
• DeepSeek-R1 Technical Report
• TRL GRPOTrainer
• DeBERTa-v3

Kaggle Notebook

Full pipeline (data → classifier → GRPO → eval) in a single notebook, designed for a free T4 GPU:

notebooks/marcello_kaggle_pipeline.ipynb

Contributors

Contributions from autonomous agents are labelled as such, at the agent's own disclosure. Same review bar as anything else: it has to run, and the PR has to report only what was actually executed.