Superintelliagent_ContinualDPO

Continual DPO agent for SDXL with per-session inference logging and periodic LoRA updates.

• Online data collection → DPO pairs → periodic fine-tuning
• Per-session “served images” logging to visualize quality over time
• Optional LoRA merge into UNet after each training round for faster inference

---

Installation

create & activate a virtual environment (example with venv)
python -m venv .venv
source .venv/bin/activate            # Windows: .\.venv\Scripts\activate

install in editable/dev mode

pip install -e .

Configure OpenAI

export OPENAI_API_KEY=sk-...         # put your real key

Prefer not to install the CLI? You can run with python -m continual_dpo ....

Quickstart

Run the demo over a JSONL file where each line includes a prompt: Async training:

export CDPO_ASYNC=1

(0 for Sync training)

Superintelliagent_ContinualDPO demo \
  --jsonl /path/to/evaluation_metadata.jsonl \
  --pretrained stabilityai/stable-diffusion-xl-base-1.0 \
  --save-dir ./nano_out \
  --store ./continual_dpo_store \
  --sessions ./sessions \
  --limit 554

Pipeline overview

1. SDXL generates from each prompt.

2. GPT-4o validates (yes/no QA) and o1 refines prompts when needed.

3. Negative/positive pairs are collected for DPO (with redundancy filtering).

4. Every 128 pairs (configurable), a DPO fine-tune runs and writes unet_lora.pt.

5. The new LoRA is (optionally) merged/loaded for the next session.

Outputs

./sessions/session_0000/inference/*.jpg — images actually served in that session

./continual_dpo_store/hfds — Hugging Face dataset used for DPO training

./continual_dpo_store/pairs.jsonl — logged DPO pairs + metadata

./nano_out/unet_lora.pt — LoRA weights after each training round

CLI Commands

Demo

Superintelliagent_ContinualDPO demo --jsonl /path/to/evaluation_metadata.jsonl --limit 554

Key options

--pretrained : base model repo/path (default SDXL base)

--save-dir : where to write LoRA weights (unet_lora.pt)

--store : where to store HF dataset & pairs JSONL

--sessions : per-session served images

--limit : how many JSONL lines to process (0 = all)

One-off inference

Superintelliagent_ContinualDPO infer \
  --prompt "a photo of a blue pizza and a yellow baseball glove"

Saves the generated image in the current session and ./demo_out/single.jpg.

Train once (existing dataset)

Superintelliagent_ContinualDPO train-once --dataset ./continual_dpo_store/hfds

Runs a single DPO round on the prepared HF dataset and writes a new unet_lora.pt.

Troubleshooting

Superintelliagent_ContinualDPO: command not found

pip install -e .
which Superintelliagent_ContinualDPO        # macOS/Linux
or
where Superintelliagent_ContinualDPO        # Windows

Or run without installing the CLI:

python -m continual_dpo demo --jsonl /path/to/evaluation_metadata.jsonl

OpenAI errors

echo $OPENAI_API_KEY        # ensure it’s set

Requirements

Python ≥ 3.9 NVIDIA GPU recommended (CUDA) for SDXL inference/training

Dependencies are installed via pip install -e . (see pyproject.toml)

Notes

Session rotation happens after each training round, so you can compare results across sessions.

LoRA merging improves inference speed but makes continued incremental training less convenient; keep it disabled during online learning if you plan frequent updates.