😃😍🔥🔥 Segmentation-Based Parametric Painting 🔥🔥😍😃

This repository contains a semantic-based painting optimization algorithm which aims to transform a given input image into a painting. The algorithm takes advantage of modern computer vision techniques, segmentation networks, and a differentiable renderer to generate results.

Project Website

What It Does

The algorithm aims to efficiently optimize a collection of stroke parameters to create a painting from a given image input. The method starts with an image and optimizes a parameter vector of strokes. When rendered on a canvas, it recreates the input image in the form of a painting. The approach is designed to efficiently manage any input size or aspect ratio. It divides the canvas into semantic areas using a segmentation network. This provides a higher control over the painting compared to previous optimization and neural methods.

• Layered Painting & Patch-Based Approach:

  • Implements a coarse-to-fine progression.
  • Uses a layered approach, starting with a rough first coarse painting pass and progressively refines.
  • Uses patches of 128x128 size and batch-optimizes all stroke parameters in each patch.

• Semantic Segmentation:

  • Provides precision over the granularity of each semantic zone in the artwork.
  • Ensures strokes remain within the designated semantic segment, improving the painting's accuracy.

• Visual Working Memory:

  • Uses a dynamic attention maps system to focus on areas that need more attention.
  • Yields an organic feel to the painting.

• Stroke Initialization, Renderer, and Blending:

  • Uses the stroke parameter representation and differentiable renderer.
  • Strokes are parameterized by a 13-dimensional tuple encoding various properties like start, middle, end points, radii, transparency, and RGB color.
  • Strokes are composited into patches with soft blending.

• Optimization & Loss Functions:

  • Optimizes all stroke parameters in batch for efficiency.
  • Uses both pixel loss and perceptual loss to ensure accurate recreation of the input image.

How To Use

1. Requirements:

• Python 3.9 or later
• Install the required packages from requirements.txt using the following command (if using Ubuntu):

pip install -r requirements.txt

• For Mac OS with Apple Silicon (M1/M2/M3) - OPTIMIZED VERSION:

pip install -r requirements_os.txt

Note: The requirements_os.txt has been updated with all necessary dependencies and is fully optimized for Apple Silicon Macs with MPS (Metal Performance Shaders) support.

• macOS Setup (Automatic): The code now automatically detects and configures MPS device support. No manual changes needed!

• Download the renderer and perceptual network here and store them in a folder under the main directory. *Update: also provided in folder model_checkpoints.

🚀 NEW PERFORMANCE OPTIMIZATIONS (2024)

This version includes significant performance and memory optimizations:

• Mixed Precision Training: Reduces memory usage by ~40%
• Early Termination: Stops optimization when loss plateaus (20-50% speedup)
• Aggressive Patch Filtering: Only processes areas that need refinement
• Memory Management: Automatic cleanup prevents memory accumulation
• MPS Optimization: Full Apple Silicon support with Metal Performance Shaders
• Image Resizing: Built-in support for processing smaller images

2. Command:

python main.py [options]

3. Key Arguments:

Core Settings:

• --image_path: Path to input image (default: 'images/paris2.jpeg')
• --style: Painting style - 'realistic', 'painterly', 'abstract', 'expressionist' (default: 'expressionist')
• --canvas_size: Canvas size for patches (default: 128, use 16-32 for faster processing)
• --save_dir: Directory to save results (default: './results')

🆕 New Optimization Arguments:

• --use_mixed_precision: Enable mixed precision training (default: True)
• --image_resize_factor: Resize input image by factor (e.g., 0.33 for 3x smaller, 0.5 for 2x smaller)
• --upsample: Enable upsampling of small images (default: True, set to False to keep resized images small)
• --min_image_size: Minimum image size for upsampling (default: 1800, set lower to avoid upsampling after resize)
• --texturize: Use texturize feature (default: True, set to False for memory savings)
• --save_animation: Save animation frames (default: True, set to False for memory savings)

Performance Settings:

• --lr: Learning rate (default: 0.004, try 0.008 for faster convergence)
• --aspect_ratio_downsample: Automatic downsampling factor for very large images (default: 3)
• --global_loss: Global loss strategy (default: False)
• --exp_name: Experiment name (default: 'exp_320')

Advanced Options:

• --brush_type: Brush type - 'straight' or 'curved' (default: 'curved')
• --canvas_color: Canvas background - 'black' or 'white' (default: 'black')
• --use_transparency: Enable stroke transparency (default: False)
• --stroke_init_mode: Stroke initialization - 'random' or 'grid' (default: 'grid')
• --renderer_ckpt_path: Path to renderer checkpoint (default: './model_checkpoints/renderer.pkl')

Style Transfer Options:

• --add_style: Enable CLIP-based style transfer (default: False)
• --style_prompt: Text prompt for style (default: 'Starry Night by Vincent Van Gogh')
• --style_transfer: Enable neural style transfer (default: False)
• --style_img_path: Path to style reference image (default: None)

4. Examples:

Basic usage:

python main.py --image_path images/paris2.jpeg --style painterly

Optimized for memory/speed (recommended for Apple Silicon):

python main.py \
  --image_path images/paris2.jpeg \
  --style painterly \
  --canvas_size 16 \
  --image_resize_factor 0.33 \
  --use_mixed_precision True \
  --texturize False \
  --save_animation False

For maximum memory efficiency:

export PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0
python main.py \
  --image_path images/paris2.jpeg \
  --style painterly \
  --canvas_size 8 \
  --image_resize_factor 0.5 \
  --lr 0.008

Ultra-fast testing (10x smaller image):

python main.py \
  --image_path images/paris2.jpeg \
  --style painterly \
  --canvas_size 8 \
  --image_resize_factor 0.1 \
  --upsample False \
  --lr 0.008

⚠️ Upsampling Fix: When using --image_resize_factor, the algorithm now automatically skips upsampling to preserve your intended image size. Use --upsample False to completely disable upsampling, or adjust --min_image_size to control the upsampling threshold.

🛠️ Troubleshooting

Memory Issues:

• Use smaller --canvas_size (8, 16, 32)
• Enable --image_resize_factor 0.5 or smaller
• Set --texturize False and --save_animation False
• Use export PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 on macOS

Performance Tips:

• Higher --lr (0.008) for faster convergence
• Early termination automatically stops when loss plateaus
• Patch filtering reduces unnecessary computations
• Mixed precision is enabled by default for optimal performance

Common Issues:

• "MPS out of memory": Reduce image size or canvas size
• "Module not found": Install missing packages with pip install <package>
• Slow performance: Use optimized arguments from examples above

📊 Performance Improvements

Before vs After Optimizations:

Optimization Features:

• 🧠 Smart Early Termination: Automatically stops when loss plateaus
• 🎯 Patch Filtering: Only processes areas that need improvement
• ⚡ Mixed Precision: Reduces memory usage without quality loss
• 🍎 Apple Silicon Optimized: Native MPS support for M1/M2/M3 chips
• 🔧 Memory Management: Automatic cleanup prevents memory leaks
• 📏 Image Resizing: Process smaller images for faster results

Method Overview

The method uses various techniques and algorithms to produce a painting from an input image. Key components include:

• Semantic Segmentation: Dividing the canvas into areas of interest.
• Layered Painting: A coarse-to-fine progression.
• Visual Working Memory: A dynamic attention maps system that focus on areas that need more attention.
• Optimization & Loss Functions: Ensuring the painting closely resembles the input image.

Implementation Details:

• Optimizer: Adam with a learning rate of 0.0002.
• All painting layers are optimized for 300 iterations.
• Canvas Background: Black.
• Segmentation Network: Uses the DETR model with a CNN (ResNet) backbone followed by an encoder-decoder Transformer.

More Results:

References

• [Huang, et al. 2019. Learning to Paint]
• [Liu et al. 2021. Paint Transformer]
• [Zou et al. 2020. Stylized Neural Painting]
• [DETR: Carion, et al. 2020. End-to-End Object Detection with Transformers]

Citation

@misc{deguevara2023segmentationbased,
      title={Segmentation-Based Parametric Painting}, 
      author={Manuel Ladron de Guevara and Matthew Fisher and Aaron Hertzmann},
      year={2023},
      eprint={2311.14271},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      }

Feedback

For any queries or feedback related to the algorithm, please open an issue on GitHub or contact the authors directly.