Evolutionary-Diffusion

Combining Evolutionary Computing with Diffusion Models

Images

• 🎨 Aesthetics Maximization/Minimization using LAION Aesthetics Predictor V2
• 📊 Multi-Objective Optimization with CLIP-IQA metrics
• 🛡️ Evading AI-image detection by optimizing against a fine-tuned SDXL detector, SSP, or DIRE
• 🧭 Navigating the CLIP-Score Landscape for Prompt-Matching

Audio

• 🔊 Optimize Audiobox Aesthetics from Meta

Try it out in Google Colab

Image results will be saved in your Google Drive in the folder evolutionary. Each generation creates a new folder where the images will be saved in. You can change the folders in the notebook.

Sometimes Google Collab causes dependency problems which break the notebook. If you have any issues executing this in a Collab environment, please do not hesitate to create a new issue.

Running locally

Using a virtual environment is recommended. Clone the repo (or download the zip), then install dependencies:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e ".[all]"

The dependency set is validated on Python 3.12 and 3.13. Now you are ready to go with the notebooks or custom code. CUDA and MPS are supported.

Package metadata now lives in pyproject.toml; setup.py is a compatibility entry point. For a locked uv environment, run uv sync --locked --extra cma for the CPU Sep-CMA example, or select --extra imaging --extra prompt_embedding --extra execution for image notebooks. See setup and extras.

Algorithm notebooks contain GA/OSGA, island GA, NSGA and Sep-CMA. Detector evaluations, campaign recipes and offline embedding analysis are organized under pre-testing.

Experiment and cloud workflows

The reusable prompt-embedding notebook uses disabled GA/OSGA examples. Install .[imaging,prompt_embedding,execution] for managed Jupyter campaigns with Python 3.11 or newer. Keep actual campaign settings and outputs under ignored .local/ or outside the checkout.

See repository architecture, campaign configuration, Azure host preparation, scientific artifacts and offline embedding analysis. The embedding bounds study documents the measured DiffusionDB/Parti comparison and its limits.

Example - Creating the most Aesthetic Image

Optimizing for Aesthetics using the Aesthetics Predictor V2 from LAION with a GA and SDXL-Turbo

Optimizing the aesthetics predictor as a maximization problem, the algorithm came to a max Aesthetics score of 8.67. This score is higher than the examples from the real LAION English Subset dataset have, with the red line showing the limit. A wide variety of prompts (inspired by parti prompts) was used for the initial population.

https://github.com/malthee/evolutionary-diffusion-results/assets/18032233/4841d671-639a-4ac4-b7a8-ee5a66fab28d

Parameters:

population_size = 100
num_generations = 200
batch_size = 1
elitism = 1

creator = SDXLPromptEmbeddingImageCreator(pipeline_factory=setup_pipeline, batch_size=batch_size, inference_steps=3)
evaluator = AestheticsImageEvaluator()  
crossover = PooledArithmeticCrossover(0.5, 0.5)
mutation_arguments = UniformGaussianMutatorArguments(mutation_rate=0.1, mutation_strength=2, clamp_range=(-900, 900)) 
mutation_arguments_pooled = UniformGaussianMutatorArguments(mutation_rate=0.1, mutation_strength=0.3, clamp_range=(-8, 8))
mutator = PooledUniformGaussianMutator(mutation_arguments, mutation_arguments_pooled)
selector = TournamentSelector(tournament_size=3)

Example - Island GA with Artists on each Island

Performing an Island GA by creating random embeddings and mixing them with artist embeddings to get mixtures of styles and new ideas.

More images

Example - Improving Audiobox Aesthetics Score

Starting from noisy random samples, evolving to better sounds. Using the sum of all fitness criteria Audiobox Aesthetics offers.

https://github.com/user-attachments/assets/25be32eb-188b-483a-a787-9cea4d7f5eeb

https://github.com/user-attachments/assets/17f4bef7-1f5a-47a4-8f29-e4f42519a4fe

Detailed Results and Notebooks

More detailed results can be found in a separate repository dedicated to the results of the experiments: https://github.com/malthee/evolutionary-diffusion-results

Evaluators

• AestheticsImageEvaluator: Uses the LAION Aesthetics Predictor V2. Blog: https://laion.ai/blog/laion-aesthetics/
• AestheticsPredictorV25ImageEvaluator: Uses the Aesthetic Predictor V2.5 from discus0434
• CLIPScoreEvaluator: Using the torchmetrics implementation for CLIP-Score
• (Single/Multi)CLIPIQAEvaluator: Using the torchmetrics implementation for CLIP Image Quality Assessment.
• AIDetectionImageEvaluator: Using the original Version from HuggingFace, or the fine-tuned one for SDXL generated images
• SSPAIDetectionImageEvaluator: Implements SSP from the paper with references from the official repo.
• DIREAIDetectionImageEvaluator: Implements DIRE from the paper with an official ADM/DDIM backend and an explicit SDXL-Turbo experimental comparison mode.
• AudioboxAestheticsEvaluator: Using Audiobox Aesthetics from Meta

Image Creators

Current supported creators working in the prompt embedding space:

• SDXLPromptEmbeddingImageCreator: Supports the SDXL pipeline, creates both prompt- and pooled-prompt-embeddings.
• SDPromptEmbeddingImageCreator: Only has prompt-embeddings, is faster but produces less quality results than SDXL.

Audio Creators

Supporting only AudioLDM because it works simply on the CLAP embedding space (suitable for this kind of operation). Other embeddings have shown to not work well with evolutionary operations (like the T5 encoder for example)

• AudioLDMSoundCreator: Works with any AudioLDMPipeline, default is audioldm-l-full

Package Structure and Base Classes

(Pre-Testing) Evaluating Models for Evolutionary use

There are multiple notebooks exploring the speed and quality of models for generation and fitness-evaluation. These notebooks also allow for simple inference so that any model can be tried out easily.

• diffusion_model_comparison: tries out different diffusion models with varying arguments (inference steps, batch size) to find out the optimal model for image generation in an evolutionary context (generation speed & quality)
• clip_evaluators: uses torch metrics with CLIPScore and CLIP IQA. CLIPScore could define the fitness for "prompt fulfillment" or "image alignment" while CLIP IQA has many possible metrics like "quality, brightness, happiness..."
• ai_detection_evaluator: evaluates AI-detection fitness for evasion using the SDXL detector, SSP, and DIRE.
• aesthetics_evaluator: uses a pre-trained model from the maintainers of the LAION image dataset, which scores an image 0-10 depending on how "aesthetic" it is. Could be used as a maximization criteria for the fitness of images.
• clamp_range: testing the usual prompt-embedding min and max values for different models, so that a CLAMP range can be set in the mutator for example. Using the parti prompts.
• crossover_mutation_experiments: testing different crossover and mutation strategies to see how they work in the prompt embedding space
• embedding_relations: recording embeddings for offline analysis

Secured auto-download of models and TLS certificates

Sadly many of these models are hosted somewhere without a proper versioning and release system.
Thus we added some hardcoded hashes of the models we use in the code, so that if the auto-download is used, it will be verified against the expected hash. You can enable/disable this verification in the Evaluator usage code.

Furthermore, the Model/checkpoint auto-download uses verified TLS. If you hit CERTIFICATE_VERIFY_FAILED on macOS Python.org builds, run:

/Applications/Python\ 3.13/Install\ Certificates.command

Alternative: set SSL_CERT_FILE to a valid CA bundle path (for example from certifi). An insecure fallback exists only via explicit opt-in for temporary troubleshooting:

export EVOLUTIONARY_DIFFUSION_ALLOW_INSECURE_SSL=1

Tests (newly added, to be extended)

pytest tests -q

Offspring selection and operator pools

See configuration, behavior and tracing and the reproducible OSGA notebook.

Classical image fitness and embedding ranges

The classical image fitness evaluators provide CPU measures with explicit scoring conventions, feasibility checks and diagnostics. The SDXL range tools compare observed token and pooled embedding bounds across pinned prompt datasets. Generated measurements and private study controllers remain local. Running the range tools performs text encoding; installation does not run experiments or change packaged clamp tensors.