Fashion Recommendation System

An AI-powered fashion recommendation engine built on a 7-layer intelligent architecture, combining computer vision, style intelligence, contextual awareness, and personalized user profiling.

🏗️ Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                    Layer 6: Virtual Try-On                      │
│           (CatVTON, Replicate AI clothing overlay)              │
├─────────────────────────────────────────────────────────────────┤
│                   Layer 5: Visualization                        │
│              (Outfit rendering & composition)                   │
├─────────────────────────────────────────────────────────────────┤
│                    Layer 4: LLM Interface                       │
│       (Explanations, Conversations, Styling Advice)             │
├─────────────────────────────────────────────────────────────────┤
│                  Layer 3: Context Engine                        │
│   (Weather, Occasion, User Profile, Body Measurements)          │
├─────────────────────────────────────────────────────────────────┤
│              Layer 2: Style Intelligence Model                  │
│      (11+ Scoring Criteria - Compatibility Analysis)            │
├─────────────────────────────────────────────────────────────────┤
│              Layer 1: Vision & Attribute Extraction             │
│        (Gemini Vision AI, Color/Pattern Analysis)               │
├─────────────────────────────────────────────────────────────────┤
│              Layer 0: Garment Segmentation                      │
│         (SAM, GroundingDINO, SCHP Human Parsing)                │
└─────────────────────────────────────────────────────────────────┘

---

📁 Project Structure

fashion_recommender/
├── src/
│   ├── layer0_segmentation/    # Garment extraction from images
│   ├── layer1_vision/          # Vision AI & Attribute extraction
│   ├── layer2_style/           # Style Intelligence & Scoring
│   ├── layer3_context/         # Context Engine & User Profiling
│   ├── layer4_llm/             # LLM Explanations & Conversations
│   ├── layer5_visualization/   # Outfit image generation
│   ├── layer6_tryon/           # Virtual try-on
│   ├── api/                    # REST API endpoints
│   ├── core/                   # Shared utilities & models
│   └── main.py
├── config/
│   ├── scoring_config.json     # Scoring profiles & weights
│   ├── data/                   # JSON rules (colors, weather, etc.)
│   ├── parameters/             # Model & conversation settings
│   └── prompts/                # LLM prompt templates
├── data/
│   ├── raw/                    # Raw fashion images
│   ├── processed/              # Processed embeddings
│   ├── sample_wardrobe/        # Sample wardrobe for testing
│   └── runway/                 # Runway looks for training
├── models/
│   ├── compatibility/          # Trained compatibility models
│   └── embeddings/             # Cached embeddings
├── scripts/
│   ├── test_full_pipeline.py   # End-to-end pipeline testing
│   └── pipeline/               # Pipeline modules
├── tests/
│   ├── unit/                   # Unit tests (670+ tests)
│   ├── integration/            # Integration tests
│   └── e2e/                    # End-to-end tests
└── notebooks/                  # Experimentation

---

🎯 Layer Details

Layer 0: Garment Segmentation

Extracts individual garments from outfit images using state-of-the-art segmentation models.

Features:

• Multi-model ensemble (GroundingDINO + SAM + SCHP)
• Precise garment boundary detection
• Category-aware segmentation
• Mask refinement and post-processing

---

Layer 1: Vision & Attribute Extraction

Analyzes garment images to extract detailed attributes using Google Gemini Vision AI.

Extracted Attributes:

GarmentAttributes:
  ├── category          # top, bottom, shoes, outerwear, accessories
  ├── subcategory       # t-shirt, jeans, sneakers, etc.
  ├── color             # ColorProfile (primary, secondary, hex codes)
  ├── pattern           # PatternInfo (type, scale, density)
  ├── material          # fabric type
  ├── fit               # slim, regular, loose, oversized
  ├── formality_level   # casual, smart_casual, formal
  ├── season            # spring, summer, fall, winter
  └── style_tags        # minimalist, bohemian, streetwear, etc.

---

Layer 2: Style Intelligence Model ⭐ (Core Scoring Engine)

The heart of the system - 11+ scoring criteria that evaluate outfit compatibility using fashion design principles.

📊 Scoring Criteria

*Weights shown for "default" profile - varies by scoring profile

🎨 Scoring Profiles

Key Modules

Scoring Output Example

🎯 Overall Score: 78.5%
📝 Grade: B+

📊 Criteria evaluated (9):
   7-Point Rule              [████████░░] 80.0%
   Color Harmony             [█████████░] 90.0%
   3-Color Rule              [██████████] 100.0%
   Proportions               [███████░░░] 70.0%
   Volume Balance            [████████░░] 83.3%
   Pattern Mixing            [█████████░] 90.0%
   Design Principles         [███████░░░] 73.0%
   Total Style               [███████░░░] 65.0%
   Creativity                [████░░░░░░] 45.0%

---

Layer 3: Context Engine

Adapts outfit recommendations based on user context, environment, and personal characteristics.

🌍 Context Criteria

👤 User Profile Pipeline

Extracts comprehensive user style profile from a photo:

StyleProfile:
  ├── body_metrics           # BMI, frame size, proportions
  │   ├── height_cm
  │   ├── weight_kg
  │   ├── bmi / bmi_category
  │   ├── frame_size         # small, medium, large
  │   ├── torso_leg_ratio
  │   ├── shoulder_hip_ratio
  │   ├── estimated_top_size
  │   └── estimated_bottom_size
  │
  ├── skin_analysis
  │   ├── skin_tone          # fair, light, medium, tan, deep
  │   └── undertone          # warm, cool, neutral
  │
  ├── hair_analysis
  │   ├── hair_color         # blonde, brown, black, red, gray
  │   └── hair_tone          # warm, cool, neutral
  │
  ├── contrast_level         # low, medium, high, very_high
  └── color_season           # SPRING, SUMMER, AUTUMN, WINTER

Key Modules

Context-Aware Scoring Output

🎯 Applying Context-Aware Scoring
============================================================
   👕 Fit Score: 0.85 (good)
   📐 Proportion Score: 0.78 (good)
   🎨 Color Harmony Score: 0.92 (excellent)

   ⭐ Overall Context Score: 0.85
   🔄 Blended Score (60% style + 40% context): 0.81

---

Layer 4: LLM Interface

Natural language explanations and conversational styling advice using LLM models.

Features:

• Natural language outfit explanations
• Personalized styling tips
• Conversational Q&A about fashion
• Tone adaptation based on user preferences

Prompt Templates:

• llm_explain_outfit_system.md - System prompt for explanations
• llm_outfit_comparison.md - Compare two outfits
• llm_styling_tip.md - Generate styling tips
• llm_personalize_message.md - Personalize recommendations
• conversation_system.md - Conversational system prompt

---

Layer 5: Visualization

Generates visual representations of outfit recommendations.

Features:

• Outfit collage generation
• Score overlay visualization
• Before/after comparisons

---

Layer 6: Virtual Try-On

AI-powered virtual try-on using state-of-the-art models.

Features:

• Realistic clothing overlay on user photos
• Multiple backend support (CatVTON, Replicate)
• Body-aware garment placement

---

🚀 Quick Start

# 1. Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# 2. Install dependencies
pip install -r requirements.txt

# 3. Set up environment variables
cp .env.example .env
# Edit .env with your API keys (GOOGLE_API_KEY, etc.)

# 4. Run the full pipeline test
python scripts/test_full_pipeline.py --demo

# 5. Test with a wardrobe folder
python scripts/test_full_pipeline.py --wardrobe ./data/sample_wardrobe/

# 6. Test with user profile (context-aware scoring)
python scripts/test_full_pipeline.py \
  --wardrobe ./data/sample_wardrobe/ \
  --user-photo ./my_photo.jpg \
  --height 175 \
  --weight 70

# 7. Run the API
uvicorn src.main:app --reload

---

🧪 Testing

# Run all unit tests (670+ tests)
pytest tests/unit/ -v

# Run specific layer tests
pytest tests/unit/layer2/ -v  # Style scoring tests
pytest tests/unit/layer3/ -v  # Context engine tests

# Run with coverage
pytest tests/unit/ --cov=src --cov-report=html

---

⚙️ Configuration

Scoring Configuration (config/scoring_config.json)

{
  "scoring_profiles": {
    "default": {
      "criteria": {
        "seven_point": { "enabled": true, "weight": 0.15 },
        "color_harmony": { "enabled": true, "weight": 0.15 },
        "three_color": { "enabled": true, "weight": 0.10 },
        "proportion": { "enabled": true, "weight": 0.15 },
        "volume_balance": { "enabled": true, "weight": 0.10 },
        "pattern_mixing": { "enabled": true, "weight": 0.10 },
        "design_principles": { "enabled": true, "weight": 0.15 },
        "creativity": { "enabled": true, "weight": 0.10 }
      }
    }
  }
}

Data Configuration (config/data/)

---

🔑 Environment Variables

# Required
GOOGLE_API_KEY=your_gemini_api_key

# Optional
OPENAI_API_KEY=your_openai_key
WEATHER_API_KEY=your_weather_api_key
REPLICATE_API_TOKEN=your_replicate_token
DATABASE_URL=your_database_url

---

📊 API Endpoints

---

📈 Scoring Algorithm Deep Dive

7-Point Rule

Based on the fashion principle that a well-balanced outfit has a point value between 4-7:

Scoring:

• 4-7 points → High score (optimal range)
• <4 points → Too basic, needs visual interest

• 7 points → Too busy, competing elements

Color Harmony

Analyzes outfit colors using color wheel relationships:

Design Principles

Based on fashion school fundamentals:

1. Balance - Visual weight distribution (symmetric/asymmetric)
2. Proportion - Size relationships (golden ratio, rule of thirds)
3. Rhythm - Repetition and flow of design elements
4. Emphasis - Focal point creation
5. Harmony - Unity of all elements
6. Contrast - Intentional differences for interest

---

📄 License

MIT License

---

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Run tests (pytest tests/unit/)
4. Commit changes (git commit -m 'Add amazing feature')
5. Push to branch (git push origin feature/amazing-feature)
6. Open a Pull Request