Special Application Models — Fine-Tuning Gemma 3/4 on 26 NLP Tasks

Specialized versions of Google Gemma 3 (and Gemma 4) trained on individual NLP tasks, with systematic comparison against the base model.

Model License: Gemma 1/2/3 → Gemma Terms of Use · Gemma 4 → Apache 2.0
Code License: Apache 2.0

---

Objective

Each folder in this repository contains:

1. A fine-tuning script (LoRA/QLoRA) for a specific NLP task.
2. The dataset used (HuggingFace link or download/preparation script).
3. An evaluation notebook comparing the fine-tuned model against the corresponding Gemma base model.

---

Supported Hardware

---

Base LoRA Configuration

r             = 16
alpha         = 16
dropout       = 0.05
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                  "gate_proj", "up_proj", "down_proj"]
lr            = 2e-4
scheduler     = "cosine"
warmup        = 0.03   # 3% of total steps

---

Critical Gemma 3 Gotchas

• fp16 NaN: activations can exceed 65,504 — use Unsloth or bf16 on Ampere+ GPUs.
• Chat template: <start_of_turn>user…<end_of_turn><start_of_turn>model…<end_of_turn>. Gemma 3 has no system role; Gemma 4 introduces it.
• attn_implementation="eager" required for Gemma 3 — SDPA/FA2 cause numerical degradation.
• Pad token: always set tokenizer.pad_token = tokenizer.eos_token to avoid silent loss masking.

---

Implemented Tasks

Text Understanding (Tasks 1–5)

Generation (Tasks 6–12)

Structured Extraction (Tasks 13–20)

Dialogue & Alignment (Tasks 21–25)

Advanced Extraction (Task 26)

---

Key Italian Datasets

SFT Note: use a 70% IT / 30% EN data mix to avoid catastrophic forgetting in Italian.

---

Repository Structure

Special Application Models/
├── README.md
├── LICENSE
├── requirements.txt
├── utils/
│   ├── base_trainer.py     ← shared LoRA/Unsloth utilities
│   └── dry_run.py          ← offline pipeline verifier
├── .github/
│   └── workflows/
│       ├── ci.yml          ← automatic dry run on every push
│       ├── release.yml     ← tag completed tasks
│       └── notify.yml      ← training completion notifications
├── task_01_sentiment/
│   ├── train.py
│   ├── dataset_prep.py
│   └── evaluate.ipynb
├── task_02_intent/
│   └── ...
└── task_26_doc_segmentation/
    └── ...

---

How to Use

1 — Repository Setup on GitHub

# From the project folder on your PC
git init
git add .
git commit -m "feat: Gemma 3/4 fine-tuning project across 26 NLP tasks"

# Create a repo on github.com, then:
git remote add origin https://github.com/<your-username>/special-application-models.git
git push -u origin main

2 — Training on Google Colab

Open a new notebook at colab.research.google.com, select a GPU (Runtime → Change runtime type → T4/L4/A100) and paste:

# Clone the project
!git clone https://github.com/<your-username>/special-application-models.git
%cd special-application-models

# Install Unsloth (includes fp16 patch for Gemma 3)
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install trl datasets evaluate rouge_score bert_score sacrebleu

# Run training for the desired task
!python task_01_sentiment/train.py --model_size 1b --num_epochs 3 --batch_size 8

# Evaluate
!jupyter nbconvert --to notebook --execute task_01_sentiment/evaluate.ipynb

To pull the latest code after local changes, just run !git pull before launching.

3 — Training on Kaggle

1. Go to kaggle.com → Notebooks → New Notebook
2. In the right panel, enable GPU T4 x2 or P100
3. In a cell at the top:

# Kaggle already has PyTorch and CUDA; install only extra dependencies
!pip install "unsloth[kaggle-new] @ git+https://github.com/unslothai/unsloth.git" trl datasets

!git clone https://github.com/<your-username>/special-application-models.git
%cd special-application-models

!python task_01_sentiment/train.py --model_size 1b --bf16

Checkpoints are saved to /kaggle/working/checkpoints/ and downloadable from the Kaggle UI.

---

GitHub Actions — CI and Automation

The repository includes three workflows in .github/workflows/:

ci.yml — Automatic check on every push

Runs syntax checking of all Python scripts and a dry run of every task on CPU, without GPU. Triggers on every git push and every Pull Request.

# .github/workflows/ci.yml
name: CI — Dry Run all tasks

on: [push, pull_request]

jobs:
  dry-run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.10"

      - name: Install lightweight dependencies (no GPU)
        run: pip install datasets transformers trl accelerate tokenizers

      - name: Python syntax check
        run: |
          python -m py_compile utils/base_trainer.py utils/dry_run.py
          for f in task_*/train.py task_*/dataset_prep.py; do
            python -m py_compile "$f" && echo "✓ $f"
          done

      - name: Dry run all tasks (CPU, synthetic data)
        run: |
          for tid in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26; do
            python utils/dry_run.py --task "$tid" --max_steps 5 --output_dir /tmp/checkpoints
          done

release.yml — Automatic tag on task completion

When you push a tag task-NN-done (e.g., git tag task-01-done && git push --tags), it automatically creates a GitHub Release with attached log files.

# .github/workflows/release.yml
name: Release — Task completed

on:
  push:
    tags:
      - "task-*-done"

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Create Release
        uses: softprops/action-gh-release@v2
        with:
          name: "✅ ${{ github.ref_name }}"
          body: "Fine-tuning completed for task ${{ github.ref_name }}."
          generate_release_notes: true

notify.yml — Email/Slack notification when training ends

You can add a webhook (e.g., Slack or email via SendGrid) to receive a notification when a Colab training run finishes. At the end of the training script in Colab, add:

# At the end of train.py, notify via GitHub Actions dispatch
import requests, os
requests.post(
    "https://api.github.com/repos/<your-username>/special-application-models/dispatches",
    headers={"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}",
             "Accept": "application/vnd.github+json"},
    json={"event_type": "training-done", "client_payload": {"task": "01"}}
)

How to create the workflow files

mkdir -p .github/workflows
# Copy the YAML contents shown above into their respective files, then:
git add .github/
git commit -m "ci: add GitHub Actions workflows"
git push

After pushing, go to github.com → your repo → Actions to see live run status.

---

References

• Gemma 3 Technical Report
• Unsloth
• MLX-LM
• LLaMA Factory
• EvalPlus (code generation)
• RAGAS (RAG evaluation)