Bridge Damage VLM Fine-Tuning: Automated Assessment System v1.0.0

Progressive Fine-Tuning of LLaVA-1.5-7B with Quality Guard Agent for Bridge Damage Analysis and Repair Priority Scoring

🌏 日本語版ドキュメント (Japanese Documentation)

📋 Table of Contents

• Overview
• Pipeline Architecture
• v0.1 Achievements
• v0.2: Quantization Comparison
• v0.3: Dataset Preparation & Fine-Tuning
• v0.4: QLoRA Progressive Training
• v0.5.1: Model Evaluation & Visualization
• v0.6.3: Quality Guard Agent & Full Pipeline
• v0.7.1: Image Size Optimization for QLoRA
• v0.7.2: pairdata_v2 Progressive Retraining (1k-4k)
• v0.7.3: Quality Guard Recalibration & Full Pipeline
• v0.7.4: 2048-Dimensional Composite Embedding Evaluation
• v0.7.5: 2k Pipeline Evaluation & Supplementary Paper Update
• v0.8: pairdata_v3 QLoRA Rank Ablation (r=16/32/64)
• v0.8.1: DoRA vs. LoRA Comparison (r=64)
• v0.8.2: Embedding Evaluation + Quality Guard Assessment → LoRA r=32 Adopted
• v0.8.3: U.7 Qualitative Analysis Full Update (pairdata v3)
• v0.8.4: DoRA Rank Ablation Study (r=16/32 vs LoRA r=32)
• v0.8.5: DoRA r=24 Final Rank Ablation & v1.0.0 Model Selection
• v1.0.0: Production Full Inference (n=10,903)
• Performance Metrics
• Setup
• Usage
• Tech Stack
• Directory Structure
• Troubleshooting
• Roadmap

---

Overview

An end-to-end pipeline for automated analysis of bridge structural damage (rebar exposure, cracks, corrosion) using LLaVA (Large Language and Vision Assistant). The system generates expert-level damage descriptions from images and produces structured prioritization scores for repair planning.

Key Features

• Multi-Modal Vision Analysis: Leverages LLaVA-1.5-7B for accurate damage assessment
• Automated Structuring: Converts natural language descriptions to structured JSON using Swallow-8B (Japanese LLM)
• Intelligent Scoring: Rule-based prioritization system (1-5 scale)
• Production-Ready: 100% success rate on 10-image test batch
• GPU-Optimized: Full GPU acceleration with quantized GGUF models (4GB)

---

Pipeline Architecture

High-Level Flow

graph TB
    A[Input Images<br/>254 Bridge Damage Photos] --> B[Preprocessing<br/>Denoise/Resize/Contrast]
    B --> C[Vision Analysis<br/>LLaVA-1.5-7B GGUF]
    C --> D[JSON Structuring<br/>Swallow-8B]
    D --> E[Priority Scoring<br/>Rule-Based Engine]
    E --> F[Output<br/>CSV/JSON Reports]
  
    style A fill:#e1f5ff
    style C fill:#fff4e1
    style E fill:#ffe1e1
    style F fill:#e1ffe1

Detailed Pipeline Components

flowchart LR
    subgraph Input["📁 Input Stage"]
        A1[Raw Images<br/>PNG/JPG]
    end
  
    subgraph Preprocess["🔧 Preprocessing"]
        B1[Noise Reduction<br/>NL-Means]
        B2[Resize<br/>Max 1024x1024]
        B3[Contrast<br/>CLAHE]
    end
  
    subgraph Vision["👁️ Vision Analysis"]
        C1[Image Encoding<br/>CLIP Vision]
        C2[LLaVA Inference<br/>Q4_K_M Quantized]
        C3[Damage Description<br/>Japanese Text]
    end
  
    subgraph Structure["📊 Structuring"]
        D1[Swallow-8B LLM<br/>via Ollama]
        D2[JSON Parsing<br/>Multi-Pattern]
        D3[Structured Data<br/>Type/Severity/Location]
    end
  
    subgraph Score["🎯 Scoring"]
        E1[Rule Evaluation<br/>40% Severity<br/>35% Type<br/>15% Location<br/>10% Risk]
        E2[Priority Calculation<br/>0.0-1.0 Score]
        E3[Level Assignment<br/>1-5 Priority]
    end
  
    subgraph Output["💾 Output"]
        F1[CSV Export<br/>Batch Results]
        F2[JSON Export<br/>Detailed Records]
    end
  
    A1 --> B1 --> B2 --> B3
    B3 --> C1 --> C2 --> C3
    C3 --> D1 --> D2 --> D3
    D3 --> E1 --> E2 --> E3
    E3 --> F1
    E3 --> F2
  
    style Input fill:#e1f5ff
    style Vision fill:#fff4e1
    style Score fill:#ffe1e1
    style Output fill:#e1ffe1

Component Details

Processing Time Breakdown (per image):

┌─────────────────────────────────────────┐
│ Preprocessing:     ~2s  (4%)            │
│ Vision Analysis:  ~42s  (81%)           │
│ JSON Structuring:  ~5s  (10%)           │
│ Scoring:          <1s   (2%)            │
├─────────────────────────────────────────┤
│ Total: ~51.6 seconds/image              │
└─────────────────────────────────────────┘

---

v0.1 Achievements

✅ Completed Features

• 3 Vision Modes Implemented

  • llama-cpp-python + GGUF (Recommended): Lightweight, fast, full GPU utilization
  • HuggingFace Transformers: Stable, high accuracy
  • Ollama Integration: Easy setup (Note: CPU-only, slower)

• Complete Pipeline

  • Preprocessing module (OpenCV)
  • Vision analysis (LLaVA-1.5-7B)
  • JSON structuring (Swallow-8B via Ollama)
  • Priority scoring (Rule-based)

• Validation Tests

  • ✅ Single image: 42s/image
  • ✅ 10-image batch: 51.6s/image avg, 100% success rate
  • Priority distribution: Critical (Level 5) 60%, Moderate (Level 3) 40%

• Windows Encoding Issues Resolved

  • PowerShell cp932 support
  • llama.cpp C++ log suppression
  • UTF-8 encoding standardization

📊 Validation Data

• Dataset: 254 images of rebar exposure damage
• GPU: NVIDIA GeForce RTX 4060 Ti (16GB VRAM)
• OS: Windows 11
• Environment: Python 3.12.10 + CUDA 12.4

---

🔬 v0.2: Quantization Comparison Study

Objective

Comprehensive evaluation of LLaVA-1.5-7B quantization levels (Q4_K_M, Q5_K_M, Q8_0) to determine the optimal balance between accuracy, speed, and model size for bridge damage assessment.

Experimental Setup

• Test Dataset: 254 rebar exposure images (full dataset)
• Models Tested:
  • Q4_K_M: 4.1GB (4-bit quantization, medium)
  • Q5_K_M: 4.8GB (5-bit quantization, medium)
  • Q8_0: 7.2GB (8-bit quantization, baseline)
• Hardware: NVIDIA RTX 4060 Ti 16GB, CUDA 12.4
• Software: llama-cpp-python 0.2.90 (GPU-enabled)

Quality Evaluation Framework

Developed a 5-point quality scoring system (v0.2 design) to assess damage description completeness:

Results Summary

Performance Comparison:

• Q5_K_M vs Q4_K_M: +17.1% size, +4.5% slower, +8.5% quality ⬆️
• Q8_0 vs Q5_K_M: +50.0% size, +34.6% slower, +3.0% quality (not significant, p=0.16)

Visualization

Basic Comparison Results

Figure 1: Performance metrics across quantization levels (N=254). Shows average inference time, model size vs total time, quality score comparison, and summary statistics.

📊 English Version | Japanese Version

Statistical Analysis

Figure 2: Comprehensive statistical analysis including violin plots (quality, text length, inference time), box plots, scatter plots (quality vs text length/inference time), and detailed statistics table.

Key Findings

1. Quality Score Distribution

Violin Plot Analysis:

• Q4_K_M: Bimodal distribution (46 images at 0-1 score, 46 at 5.0) → High variance
• Q5_K_M: Concentrated at 3-4 points (75 images) → Stable mid-high quality
• Q8_0: Similar to Q5_K_M (81 images at 3-4 points) → Stable but slower

Statistical Significance (Mann-Whitney U Test):

• Q5_K_M vs Q4_K_M: U=34822.5, p=0.0591 (marginal significance)
• Q8_0 vs Q5_K_M: U=33870.0, p=0.1627 (not significant)
• Q8_0 vs Q4_K_M: U=36289.5, p=0.0069 ✓ (significant, p<0.01)

2. Text Length vs Quality Correlation

Insight: Q5_K_M maintains consistent quality regardless of description length, indicating robust performance.

3. Inference Time Analysis

• Q4_K_M: Fastest (5.43s ± 1.07s), but quality variance is high
• Q5_K_M: Slightly slower (5.67s ± 1.14s), best quality-speed balance
• Q8_0: Slowest (7.63s ± 1.42s), minimal quality improvement over Q5_K_M

Speed-Quality Efficiency:

Q4_K_M: 0.54 quality/sec (2.93 / 5.43)
Q5_K_M: 0.56 quality/sec (3.18 / 5.67) ← BEST ⭐
Q8_0:   0.43 quality/sec (3.27 / 7.63)

Discussion

Why Q5_K_M is Optimal

1. Best Quality-Speed Balance

   • Only 4.5% slower than Q4_K_M
   • 8.5% higher quality than Q4_K_M (approaching statistical significance)
   • Statistically equivalent quality to Q8_0 (p=0.16)

2. Stable Performance

   • Lowest text-length correlation (-0.148) → consistent output
   • Tight standard deviation (1.35 vs 1.53 for Q4_K_M)
   • Predictable inference time (5.67s ± 1.14s)

3. Resource Efficiency

   • 33% smaller than Q8_0 (4.8GB vs 7.2GB)
   • 25% faster than Q8_0 (34.6% speed advantage)
   • Fits comfortably in 8GB VRAM GPUs

Q8_0 Limitations

• Diminishing Returns: Only 3% quality improvement over Q5_K_M
• Slower: 34.6% longer inference time without significant quality gain
• Larger: 50% more disk space and VRAM usage
• Not Cost-Effective: Poor quality/sec efficiency (0.43 vs 0.56 for Q5_K_M)

Q4_K_M Use Cases

• Rapid Prototyping: Fastest iteration for development
• Resource-Constrained Environments: When speed is critical
• Not Recommended for Production: High quality variance (SD=1.53) creates inconsistent results

Recommendations

For Bridge Damage Assessment:

• Deploy Q5_K_M for operational use (254 images in ~24 minutes)
• Avoid Q8_0 unless accuracy requirements justify 35% slower processing
• Use Q4_K_M only for development/testing

Lessons Learned (v0.2)

1. Quantization is Not Free

   • Q4_K_M's speed advantage comes at the cost of quality variance
   • Bimodal distribution (0-1 or 5 points) indicates unstable outputs

2. Sweet Spot Exists

   • Q5_K_M achieves 97% of Q8_0's quality with 25% speed improvement
   • Middle-ground quantization often optimal for production

3. Statistical Validation is Essential

   • Correlation analysis revealed Q5_K_M's consistency advantage
   • Mann-Whitney U test confirmed Q8_0 vs Q5_K_M differences are not significant

4. GPU Compatibility Matters

   • llama-cpp-python 0.2.90 required for GPU support on Windows
   • Newer versions (0.3.x) have Visual Studio CUDA integration issues

5. Quality Metrics Enable Optimization

   • 5-point scoring framework made quantization trade-offs measurable
   • Violin plots revealed distribution differences invisible in averages

---

🎯 v0.3: Dataset Preparation & Progressive Fine-Tuning

Objective

Prepare high-quality training datasets from n=10,789 bridge damage images for progressive QLoRA fine-tuning of LLaVA-1.5-7B to improve damage description quality from 3.18/5.0 → 4.5+/5.0.

Status: ✅ Phase 1 Complete (Dataset Preparation)

Completion Date: 2026-05-22

Dataset Overview

• Source: 10,789 bridge damage images with Japanese ground truth annotations
• Master File: Rank_c_image_text_n10789.xlsx
• Quality Filtering: Applied to ensure high-quality training data
• Stratified Sampling: Maintains balanced distribution across damage scenarios
• Progressive Scaling: 1k → 2k → 3k → 4k training sets for systematic evaluation

Data Preparation Pipeline

1. Quality Filtering Criteria

Text Length: 15-500 characters
Excluded Patterns:
  - Empty or whitespace only
  - Question marks only ("???")
  - "なし" (none), "該当なし" (not applicable)
Required Keywords (at least one):
  Damage Types: ひび割れ, クラック, 鉄筋, 腐食, 剥離, 劣化
  Components: 桁, 床版, 支承, 橋脚, コンクリート
  Severity: 著しい, 顕著, 進行, 軽微

Expected Retention Rate: ~70-80% (7,600-8,600 images passing filters)

2. Stratified Sampling Strategy

Purpose: Maintain representative distribution of component types and damage types across all datasets.

Methodology:

• Strata Definition: Joint distribution of component_type × damage_type
• Proportional Allocation: Each stratum represented according to its frequency
• Fixed Random Seed: seed=42 for reproducibility

Benefits:

• ✅ Prevents model bias toward frequent categories
• ✅ Ensures test set covers all damage scenarios
• ✅ Enables fair comparison across training scales (1k/2k/3k/4k)

3. Dataset Splits

Note: Each training set is independently sampled (not cumulative) to measure pure dataset size effects.

Implementation

Script: prepare_v03_dataset.py

Features:

• Loads Excel master data (10,789 rows)
• Applies 4-stage quality filtering
• Merges v0.2 metadata for stratification (if available)
• Creates fixed test set (800 samples)
• Generates progressive training sets (1k/2k/3k/4k)
• Produces comprehensive statistics report

Usage:

# Activate virtual environment (training)
.\.venv-train\Scripts\Activate.ps1

# Run dataset preparation
python prepare_v03_dataset.py

Output Files:

data/v03_fine_tuning/
├── test_set_n800.csv              # Fixed test set
├── train_1k.csv                    # 1,000 training samples
├── train_2k.csv                    # 2,000 training samples
├── train_3k.csv                    # 3,000 training samples
├── train_4k.csv                    # 4,000 training samples
└── dataset_preparation_report.md   # Statistics & distribution analysis

CSV Format:

ファイルパス,所見,component_type,damage_type,text_length
rank_c_images_n10789/IMG_0001.jpg,"床版下面に顕著なひび割れ...",床版,ひび割れ,85

Why Progressive Training (1k→2k→3k→4k)?

Rationale

1. Measure Dataset Size Impact

   • Compare model performance at different data scales
   • Identify diminishing returns threshold
   • Determine optimal dataset size for cost-benefit

2. Detect Overfitting Threshold

   • Monitor when additional data stops improving quality
   • Observe if validation scores plateau or decline
   • Adjust regularization if needed

3. Resource Planning

   • Estimate training time per 1k samples (~2-4 hours)
   • Total time: ~20-40 hours across all 4 stages
   • Budget GPU hours for future scaling

4. Ablation Study

   • Isolate effect of data quantity vs quality
   • Validate that improvements are from data, not hyperparameters
   • Support academic publication (arXiv paper)

v0.4-v0.6 Roadmap (Next Phases)

---

🎓 v0.4: QLoRA Progressive Training

Status: ✅ Complete

Completion Date: 2026-05-23
Training Period: 2026-05-22 01:59 - 2026-05-23 00:17 (Total: ~15 hours)

Objective

Execute progressive QLoRA fine-tuning of LLaVA-1.5-7B across four dataset scales (1k, 2k, 3k, 4k samples) to measure dataset size impact on model performance and identify the optimal training data volume.

Training Results Summary

Key Findings

1. Performance Plateau at 2k-3k Range

   • Significant improvement from 1k→2k (+1.98%)
   • Plateau at 2k→3k (0% improvement)
   • Marginal improvement at 3k→4k (+0.20%)

2. Optimal Model: 2k Samples (v0.4.2)

   • Best cost-benefit ratio: 2.2x faster than 4k for only 0.2% worse loss
   • Stable training with minimal overfitting
   • Validation loss: 3.073 (vs 3.067 for 4k)

3. Diminishing Returns

   • Loss convergence confirms limited benefit beyond 2k samples
   • Training time scales approximately linearly with data size
   • Resource efficiency decreases significantly after 2k

Training Configuration

Base Model: llava-hf/llava-1.5-7b-hf
Quantization: 4-bit NF4 with double quantization

LoRA Parameters:
  rank: 32
  alpha: 64
  dropout: 0.05
  target_modules: [q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj]

Training Hyperparameters:
  batch_size: 4
  gradient_accumulation_steps: 4  # Effective batch size: 16
  learning_rate: 2e-4
  num_epochs: 3
  warmup_steps: 50
  max_grad_norm: 1.0
  weight_decay: 0.01
  optimizer: AdamW
  lr_scheduler: cosine
  mixed_precision: fp16

Hardware:
  GPU: NVIDIA GeForce RTX 4060 Ti 16GB
  CUDA: 12.4
  PyTorch: 2.6.0

Loss Convergence Visualization

Validation Loss
3.14 ┤
     │ 1k ●
3.12 ┤     ╲
     │      ╲
3.10 ┤       ╲ 2k ●━━━━━━━━━━━━●━━━━● (plateau)
     │              3k            4k
3.08 ┤
     │
3.06 ┤                              ●
     │                             4k
     └────────────────────────────────
      1k    2k    3k    4k

Model Artifacts

models/
├── llava_v03_qlora_1k/    # 134.5 MB (v0.4.1) ✅
├── llava_v03_qlora_2k/    # 134.5 MB (v0.4.2) ✅ Recommended
├── llava_v03_qlora_3k/    # 134.5 MB (v0.4.3) ✅
└── llava_v03_qlora_4k/    # 134.5 MB (v0.4.4) ✅

Detailed Analysis

See Result_QLoRA_Scale.md for:

• Detailed loss curves for all 4 stages
• Cost-benefit analysis
• Training time scaling analysis
• Validation loss stability comparison
• Recommendations for production deployment

Documentation

• Training Script: train_v03_qlora.py ✅
• Results Report: Result_QLoRA_Scale.md ✅
• Setup Guide: SETUP_VENV_VLM.md ✅
• Quick Start: QUICKSTART_V03_V05.md ✅

---

📊 v0.5.1: Model Evaluation & Visualization

Status: ✅ Complete

Completion Date: 2026-05-24
Evaluation Period: 2026-05-23 - 2026-05-24

Objective

Comprehensive evaluation of all 4 QLoRA fine-tuned models (1k/2k/3k/4k) using semantic similarity metrics and creation of publication-quality visualizations for ACVR 2026 paper submission.

Evaluation Framework

Semantic Similarity Metric (Sentence-BERT)

Replaced simple quality scoring with Japanese Sentence-BERT cosine similarity:

• Model: sonoisa/sentence-bert-base-ja-mean-tokens-v2
• Metric: Cosine similarity between model predictions and ground truth
• Range: 0.0 (completely different) → 1.0 (identical)
• Test Set: 800 samples (fixed from v0.3)

Quality Tier Classification

Evaluation Results

Cosine Similarity Performance

Quality Tier Distribution

Key Findings: Inverted-U Relationship

1. Performance Trajectory

Cosine Similarity
0.70 ┤           ╭──● 3k (Peak)
     │          ╱    ╲
0.68 ┤        ●        ╲● 4k (Degradation)
     │      ╱  2k       ╲
0.66 ┤    ╱              ╲
     │  ●                 ╲
0.64 ┤ 1k
     └──────────────────────────
      1k    2k    3k    4k

Observation: Semantic similarity improves from 1k→2k (+5.5%) → 3k (+0.9%), but degrades at 4k (-2.5%).

2. Optimal Model: 3k Samples

• Highest Mean: 0.6909 (closest to "Good" tier boundary of 0.70)
• Highest Median: 0.7081 (actually reaches "Good" tier)
• Best Good+ Ratio: 21.0% (vs 20.8% for 2k, 15.9% for 4k)
• Lowest Standard Deviation: 0.0784 (most consistent)

3. Cost-Benefit Winner: 2k Samples

• Near-Optimal Performance: 0.6850 (only -0.9% below 3k)
• Best Good+ Distribution: 20.8% Good + 0.1% Excellent
• Lowest Very Poor Rate: 2.2% (vs 5.2% for 1k)
• Training Efficiency: 2.9 hours (vs 4.5 hours for 3k, 6.3 hours for 4k)
• Recommended for Production: Best quality/time ratio

4. Overfitting at 4k

Evidence:

• 2.5% performance drop from 3k despite 33% more data
• Good tier collapses from 21.0% (3k) → 15.9% (4k)
• Poor tier expands from 27.6% (3k) → 32.1% (4k)
• Validation loss: 3.073 (2k/3k) vs 3.067 (4k) — lower loss but worse test performance

Hypothesis: Additional data introduces label noise or vocabulary saturation.

Visualizations (Publication Quality)

Created 5 comprehensive figures for ACVR 2026 paper:

Figure 3: Similarity Comparison

Bar chart comparing mean cosine similarity across all 4 models with error bars (±1σ).

Figure 4: Tier Distribution (Stacked)

100% stacked bar chart showing quality tier proportions per model.

Figure 5: Tier Distribution (Grouped)

Grouped bar chart enabling direct comparison within each quality tier.

Figure 6: Violin Plots

Distribution visualization showing density concentration (2k/3k near Good/Acceptable boundaries).

Figure 7: Summary Table

Comprehensive statistical summary with mean/std/min/max/median and tier percentages.

Paper Integration

Submitted to: ACVR 2026 (Asian Conference on Computer Vision and Robotics)
Paper: methodology.tex
Total Pages: 16 pages
Figures: 7 figures (including Validation Loss curve)

Key Sections

1. Results (§4): Quantitative evaluation with all 5 visualization figures
2. Discussion (§5):
   • §5.1: The 3k Peak and 4k Degradation (inverted-U analysis)
   • §5.2: Validation Loss interpretation (train-test discrepancy)
   • §5.3: Inference speed optimization (warm-up hypothesis)

Critical Insights for Paper

1. Loss-Based Early Stopping Limitation
   4k model achieves lowest validation loss (3.067) but worst test performance (0.6739) — highlights need for domain-specific metrics beyond perplexity.

2. Mid-Scale Training Advantage
   2k-3k range achieves optimal quality-cost balance, challenging "more data = better" assumption in specialized domains.

3. Overfitting in Low-Resource Domains
   Vocabulary saturation and label noise become dominant factors beyond ~3k samples for Japanese bridge damage terminology.

Scripts & Tools

• Visualization Creation: visualize_model_comparison.py ✅
• Inference Script: inference_v051_qlora.py ✅
• Evaluation Module: src/evaluation/vector_similarity_evaluator.py ✅

Next Steps (v0.6)

1. ACVR 2026 Submission: Complete camera-ready version with reviewer feedback
2. Production Deployment: Deploy 2k model for operational bridge inspection
3. Ablation Studies:
   • Test alternative PEFT methods (IA³, Prompt Tuning)
   • Experiment with different base models (LLaVA-v1.6, Qwen-VL)
4. Expand Test Coverage: Evaluate on different damage types (cracks, corrosion without rebar exposure)

---

v0.5-v0.6 Roadmap (Next Phases)

Phase 2: v0.4 QLoRA Fine-Tuning (⏳ Pending)

Script: train_v03_qlora.py

QLoRA Configuration:

Base Model: llava-hf/llava-1.5-7b-hf
Quantization: 4-bit NF4 with double quantization
LoRA Parameters:
  rank: 32
  alpha: 64
  dropout: 0.05
  target_modules: [q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj]
Training Hyperparameters:
  batch_size: 4
  gradient_accumulation_steps: 4  # Effective batch size: 16
  learning_rate: 2e-4
  num_epochs: 3
  optimizer: AdamW
  lr_scheduler: cosine with warmup (50 steps)
  mixed_precision: fp16

Training Stages:

# Stage 1: 1k samples (~2-4 hours)
python train_v03_qlora.py --train-data data/v03_fine_tuning/train_1k.csv --output-dir models/llava_v03_qlora_1k

# Stage 2: 2k samples (~4-8 hours)
python train_v03_qlora.py --train-data data/v03_fine_tuning/train_2k.csv --output-dir models/llava_v03_qlora_2k

# Stage 3: 3k samples (~6-12 hours)
python train_v03_qlora.py --train-data data/v03_fine_tuning/train_3k.csv --output-dir models/llava_v03_qlora_3k

# Stage 4: 4k samples (~8-16 hours)
python train_v03_qlora.py --train-data data/v03_fine_tuning/train_4k.csv --output-dir models/llava_v03_qlora_4k

Expected Output:

• LoRA adapter weights (adapter_model.safetensors)
• Training configuration (adapter_config.json)
• Training logs (TensorBoard format)
• Training summary (JSON with loss curves, timing)

Phase 3: v0.5 Vector Similarity Evaluation (⏳ Pending)

Script: src/evaluation/vector_similarity_evaluator.py

Evaluation Method:

• Model: Sentence-BERT Japanese (sonoisa/sentence-bert-base-ja-mean-tokens-v2)
• Metric: Cosine similarity between ground truth and predictions
• Test Set: Fixed 800 images (test_set_n800.csv)

Quality Thresholds:

Excellent:   ≥0.85
Good:        ≥0.75
Acceptable:  ≥0.65
Poor:        ≥0.50
Very Poor:   <0.50

Usage:

# Evaluate single model
python -m src.evaluation.vector_similarity_evaluator \
  --csv inference_1k_results.csv \
  --gt-col "所見" \
  --pred-col "prediction" \
  --output data/v03_fine_tuning/evaluations/evaluation_train_1k.json

# Generate progressive training report
python create_progressive_training_report.py \
  --eval-dir data/v03_fine_tuning/evaluations \
  --output-dir data/v03_fine_tuning/reports

Expected Improvements:

Phase 4: v0.6 Academic Paper (⏳ Pending)

Target: arXiv preprint submission

Paper Structure:

1. Abstract: Progressive fine-tuning approach for bridge damage VLM
2. Introduction: Motivation, challenges in civil engineering AI
3. Methodology:
   • Dataset construction (10,789 images → filtered → stratified)
   • QLoRA configuration (4-bit, rank=32)
   • Progressive training strategy (1k→4k)
   • Vector similarity evaluation
4. Results:
   • Quantization comparison (v0.2)
   • Fine-tuning performance scaling (v0.3-v0.5)
   • Ablation studies
5. Discussion: Optimal dataset size, cost-benefit analysis
6. Conclusion: Recommendations for domain-specific VLM deployment

LaTeX Source: paper_damage_vlm/0_Format_arXiv_pdfLaTex/bridge_damage_vlm_quantization_2026.tex

Current Status Summary

Documentation

• Setup Guide: SETUP_VENV_VLM.md - Virtual environment configuration
• Quick Start: QUICKSTART_V03_V05.md - Step-by-step workflow
• Implementation Summary: V03_V05_IMPLEMENTATION_SUMMARY.md - Technical details

---

🛡️ v0.6.3: Quality Guard Agent & Full Pipeline Evaluation

Status: ✅ Complete

Completion Date: 2026-05-25
Evaluation: Full n=800 test set (v0.6.3 pipeline, 3k model)

Overview

v0.6.3 introduces a two-stage Quality Guard Agent that filters low-quality VLM outputs before priority scoring, completing the end-to-end production pipeline:

Image → LLaVA-1.5-7B (3k QLoRA) → Quality Guard (Stage 1 + 2) → Priority Scorer → Report

Quality Guard Agent Architecture

Stage 1: Rule-Based Filter (CPU, ~0.01 s/row)

Token counting uses CJK + ASCII tokenization to correctly handle Japanese text.

Stage 2: SLM Judge (Swallow-8B NF4, GPU, Unsloth)

Applies natural-language judgement to Stage 1 PASS outputs as a second verification layer. Empirical result: Stage 2 produced 0 additional rejections for the 3k model — the rule-based Stage 1 alone was sufficient.

n=800 Evaluation Results

Priority Scoring Results (PASS n=727)

Finding — Output Mode Collapse: The 3k model generates near-identical descriptions for all PASS images. Main Girder is mentioned in 100% of predictions; Rebar Exposure in 99.6%; Spalling in 98.6%. This stereotyped output pattern causes uniform priority scores (Level 3 saturation). See paper_damage_vlm/1_Methodology/methodology.tex for full analysis.

Quality Metric Distributions

The Quality Guard preferentially retains predictions with higher semantic alignment — even though the filter is defined only on output-text properties, it acts as a proxy for semantic quality.

Key Findings

1. Stage 1 sufficiency: The lightweight CPU rule filter achieves the same classification as the full two-stage pipeline for the 3k model and this corpus.
2. Output mode collapse: Fine-tuning on a skewed training corpus causes the model to default to a dominant member–damage template. Requires data balancing or constrained-generation prompts.
3. Priority scoring calibration gap: All 727 PASS samples receive Priority Level 3, indicating the YAML rules need recalibration or VLM prompts need explicit severity keywords.
4. Throughput: 8.97 s/row (batch_size=8, torch.compile()) enables processing of 800 images in ~2 hours on RTX 4060 Ti 16GB.

Generated Figures

Scripts

• Pipeline: pipeline_v063.py — Full end-to-end v0.6.3 pipeline
• Figure generation: create_figures_10.py — Analysis figures for n=800 results
• Output CSV: data/v03_fine_tuning/evaluations/v063_scoring_results.csv (800 rows, 22 columns)

Paper Status

The academic paper (paper_damage_vlm/1_Methodology/methodology.tex) is complete with all v0.6.3 results:

• Quality Guard Agent methodology section (TikZ algorithm flow diagram)
• n=800 results tables (tab:qg_final)
• Output mode collapse analysis (fig:member_damage)
• Priority scoring distribution (fig:priority_violin)
• Discussion: Quality Guard efficacy, mode collapse, calibration
• 25 pages, 10 figures — compiled with pdflatex (no errors)

---

🖼️ v0.7.1: Image Size Optimization for QLoRA Fine-Tuning

Status: ✅ Complete

Completion Date: 2026-05-25
Experiment: Controlled comparison of 3 input image datasets at 1k scale

Background

LLaVA-1.5-7B uses a CLIP ViT-L/14-336 vision encoder pretrained at 336×336 px. Feeding images at arbitrary sizes forces internal rescaling and introduces resampling artifacts into the patch embeddings, potentially degrading fine-tuning quality. This experiment quantifies the impact of pre-standardising training images to the model's native resolution.

Image Preprocessing Pipeline

Center-crop (rather than pad-and-resize) was chosen because bridge damage photos are mixed portrait/landscape. Padding introduces black borders that divert the vision encoder's attention from damage regions.

Comparative Training Results (1k Scale, 3 Epochs)

All runs: train_v07_qlora.py, LoRA r=32/α=64, LR=2e-4, batch_size=4, BF16, RTX 4060 Ti 16GB

Key Finding

336×336 px is the optimal input size for LLaVA-1.5-7B fine-tuning on bridge damage images:

• +0.6 pt token accuracy vs. variable-size originals
• 14% shorter training time (1:34:53 → 1:21:46) due to uniform tensor shapes enabling better GPU batching
• 448×448 px shows no accuracy gain over 336×336 px, despite 76% larger file size — the CLIP encoder's 336 px pretraining is the binding constraint

Configuration Update

config.yaml and train_v07_qlora.py now default to data/inspect_images_336/:

# config.yaml — v0.7.1
v03_dataset:
  image_dir: "data/inspect_images_336"   # 336×336 center-crop (LLaVA-1.5 native)
  # image_dir: "data/inspect_images_448"  # 448×448 alternative
  # image_dir: "data/inspect_images_v2"   # max 1024px long side (original)

Scripts

• resize_images_v2.py — Step 1: long-side 1024 px resize
• crop_resize_images.py — Steps 3a/3b: center-crop + resize to 336 and 448
• train_v07_qlora.py — Added --image-base-dir CLI option for dataset switching

Further Reading

• docs/Lesson_Image_Size.md — Full lesson notes and S7 comparison table

---

📈 v0.7.2: pairdata_v2 Progressive Retraining (1k-4k)

Status: ✅ Complete

Completion Date: 2026-05-27
Scope: Full progressive QLoRA retraining on denoised pairdata_v2 with inspect_images_336

What Was Completed

• Completed all four scales (1k, 2k, 3k, 4k) under a fixed training recipe
• Updated supplementary section S.4 in methodology with final tables and observations
• Added S.4 figure using the same plotting style as Figure 9:
  • paper_damage_vlm/1_Methodology/figures_vlm/11_s4_validation_loss_pairdata_v2.png
  • plotted from raw eval checkpoints (no interpolation) for fair comparison

Final Progressive Results (pairdata_v2)

Key Findings

• Strongest gain is 1k -> 2k (-0.043 nats eval loss, +8.6 pp token accuracy)
• Clear diminishing returns beyond 2k:
  • 2k -> 3k: -0.016 nats, +0.3 pp
  • 3k -> 4k: -0.010 nats, +0.2 pp
• In 4k run, minimum eval loss appears at epoch 2 (3.124), then slightly increases at epoch 3 (3.125), suggesting near-convergence

Semantic Similarity Evaluation on Denoised Test Set (S.6)

Test set: n=800, pairdata-v2 ground truth
Evaluator: paraphrase-multilingual-MiniLM-L12-v2 (cosine similarity)
Tier thresholds: Excellent ≥ 0.85 · Good ≥ 0.70 · Acceptable ≥ 0.55 · Poor ≥ 0.40 · Very Poor < 0.40

All four models fall in the Acceptable tier under pairdata-v2 thresholds.
Absolute values are ~0.08–0.09 lower than pairdata-v1 results, reflecting the harder matching task of shorter denoised ground-truth texts.

Key findings:

• 1k → 2k: +6.3% mean gain (largest improvement)
• 2k ≈ 3k: Δρ = 0.0002 (saturation)
• 4k: −4.5% from 3k (overfitting: VPoor rises 21.5% → 28.2%)

Model selected: 3k
Rationale: peak mean (0.6013) + highest median (0.6213) + lowest Very Poor rate (21.5%).
Result is consistent with pairdata-v1 finding where 3k also achieved peak performance (ρ = 0.6909).

Figures generated:

• paper_damage_vlm/1_Methodology/figures_vlm/12_v072_similarity_by_scale.png
• paper_damage_vlm/1_Methodology/figures_vlm/13_v072_tier_distribution.png
• paper_damage_vlm/1_Methodology/figures_vlm/14_v072_cosine_violin.png

Repro Commands

.\.venv-train\Scripts\Activate.ps1 ; python -u train_v07_qlora.py --train-size 1k
.\.venv-train\Scripts\Activate.ps1 ; python -u train_v07_qlora.py --train-size 2k
.\.venv-train\Scripts\Activate.ps1 ; python -u train_v07_qlora.py --train-size 3k
.\.venv-train\Scripts\Activate.ps1 ; python -u train_v07_qlora.py --train-size 4k

Paper Artifacts Updated

• paper_damage_vlm/1_Methodology/methodology.tex (本編のみ、Supplementary分離済み)
• paper_damage_vlm/1_Methodology/methodology_supp.tex (Supplementary 独立ファイル、S.6結論含む)
• paper_damage_vlm/1_Methodology/methodology.pdf (23 pages, recompiled)
• paper_damage_vlm/1_Methodology/methodology_supp.pdf (7 pages, recompiled)

---

🔬 v0.8: pairdata_v3 QLoRA Rank Ablation

Status: ✅ Complete

Completion Date: 2026-05-30
Scope: pairdata_v3 denoising; LoRA rank ablation r=16/32/64 at 2,500-sample scale; Supplementary 2 full documentation

LoRA Rank Ablation Results (pairdata_v3, 2,500 samples)

All best checkpoints at epoch 2.421 of 3.

Key Finding

r=64 selected as optimal: 168.8M trainable parameters (4.56% of 7B backbone) provide sufficient representation space to learn the fine-grained correspondence between complex damage imagery and structured recognition text (member type × damage category). r=16/32 (42M/84M) leave a residual alignment gap. Critically, r=64 adds no measurable GPU cost over r=16 (3h24m vs 3h27m on RTX 4060 Ti 16GB).

Supplementary 2 Documentation

• U.1: Motivation (pairdata_v3 noise removal rationale)
• U.2: Extended noise pattern definition
• U.3: Pairdata v2 vs v3 statistics comparison
• U.4: QLoRA training at 2,500-sample scale — rank ablation table + Figure 21 + analysis
• U.5–U.7: Semantic similarity, Quality Guard, Qualitative analysis (pending v0.8 inference)
• U.8: DoRA vs. LoRA comparison (filled in v0.8.1)

---

🔬 v0.8.1: DoRA vs. LoRA Comparison (r=64)

Status: ✅ Complete

Completion Date: 2026-05-30
Scope: DoRA fine-tuning (use_dora=True, r=64) on pairdata_v3 2,500 samples; U.8 Supplementary 2 fully documented

DoRA vs. LoRA Results (pairdata_v3, 2,500 samples, r=64)

Best checkpoints at epoch 2.421; final-epoch: LoRA 3.132/98.93%, DoRA 3.132/98.95%.

Research Questions Answered

Supplementary 2 Updates (U.8)

• Configuration table: r=32/alpha=64 → r=64/alpha=128
• Training Results table: TBD values filled with actual DoRA results
• Motivation rewritten: r=64 basis from U.4 ablation; 3 research questions
• Adapter capacity analysis: 168.8M params at 4.56% provides bridge-domain bandwidth
• Repro Commands: updated to train_v09_dora.py --lora-r 64

Reproduce

# DoRA training
.venv-train\Scripts\Activate.ps1
python -u train_v09_dora.py --lora-r 64 *> v09_dora_r64_log.txt
# Output: models/llava_v09_dora_r64_2500

# Recompile Supplementary 2
cd paper_damage_vlm\1_Methodology
pdflatex -interaction=nonstopmode methodology_supp2.tex
pdflatex -interaction=nonstopmode methodology_supp2.tex

---

🔬 v0.8.2: Embedding Evaluation + Quality Guard Assessment

Status: ✅ Complete

Completion Date: 2026-05-31
Scope: 2048-dim Sentence-BERT evaluation of all 4 adapters on n=800 test set; Quality Guard + Priority Scoring for LoRA r=32 and DoRA r=64; Structural Member & Damage Type analysis; final model selection

Sentence-BERT Evaluation Results (2048-dim composite, n=800)

Embedding model: hotchpotch/static-embedding-japanese (1024-dim) ⊕ intfloat/multilingual-e5-large (1024-dim), L2-normalized concatenation.

Quality Guard + Priority Scoring Results (n=800)

θ_low=30, θ_high=200 (retained from v0.7.3).

Key Findings

Structural Member & Damage Type Analysis (U.6.1)

DoRA r=64 assigns exactly 3 damage types to exactly 2 members across all 798 predictions: clear evidence of fine-tuning failure.

Supplementary 2 Updates (U.5–U.6.1)

• U.5: Sentence-BERT results table filled (TBD→actual); Figures 22/23/24 inserted; Analysis paragraph added
• U.6: Quality Guard table (2-model comparison); Priority table; Analysis rewritten with v0.7.3-3k comparison and DoRA failure conclusion
• U.6.1: New subsection — Structural Member & Damage Type (Figure 28, same style as S.7.1)
• U.5 conclusion: Revised from "DoRA adopted" to "both forwarded to QG for evaluation"
• Final conclusion: LoRA r=32 (v0.8) adopted; DoRA finetuning left as future work
• methodology_supp2.pdf: 21 pages

Figures Generated

Reproduce

# Sentence-BERT evaluation (4 models)
.venv-eval\Scripts\Activate.ps1
python -u evaluate_v08_sbert.py

# Quality Guard + Priority Scoring
python -u pipeline_v08.py

# Generate Figures 22-24 (U.5)
python -u create_u8_figures.py

# Generate Figures 25-27 (U.6)
python -u create_u6_figures.py

# Generate Figure 28 (U.6.1, S.7.1 style)
python -u create_u61_figures.py

# Recompile Supplementary 2 (21 pages)
cd paper_damage_vlm\1_Methodology
pdflatex -interaction=nonstopmode methodology_supp2.tex
pdflatex -interaction=nonstopmode methodology_supp2.tex

---

🔬 v0.8.3: U.7 Qualitative Analysis Full Update (pairdata v3)

Status: ✅ Complete

Completion Date: 2026-05-31
Scope: U.7 qualitative analysis section fully rebuilt on pairdata v3 (inference_r32.csv); 12 representative PASS images re-selected with member × damage diversity; caption simplified and enriched; shared keyword highlighting added

Changes

Selected Images (12 panels, member × damage)

Reproduce

.venv-eval\Scripts\Activate.ps1
python create_u7_figures.py

cd paper_damage_vlm\1_Methodology
pdflatex -interaction=nonstopmode methodology_supp2.tex

---

🔬 v0.8.4: DoRA Rank Ablation Study

Status: ✅ Complete

Completion Date: 2026-06-01
Scope: DoRA fine-tuning at r=16 (full pipeline: train → merge → inference → SBERT → quality guard → figures); 3-way rank ablation (LoRA r=32 vs. DoRA r=32 vs. DoRA r=16) with Figures 35–41 and U.7.2 discussion update

Overview

v0.8.4 extends the DoRA investigation with a lower-rank variant (r=16) and performs a full 3-way comparison against the LoRA r=32 baseline and DoRA r=32. The ablation reveals a non-monotone relationship between DoRA rank and output quality: r=16 achieves the most balanced priority distribution despite having the smallest trainable capacity.

DoRA r=16 Training Details

3-Way Ablation Results (n=800)

SBERT Cosine Similarity (2048-dim composite evaluator)

Quality Guard & Priority Distribution (n=800)

Key finding: DoRA r=64 undergoes complete mode collapse (all Level 4). DoRA r=16 achieves the most balanced Level 3/4/5 distribution among DoRA variants.

Figures Generated

Supplementary Paper Updates

Discussion Summary

• DoRA r=64: Direction–magnitude decomposition exceeded model capacity at 2500-sample scale → priority output collapsed to Level 4 (100%), losing member/damage diversity
• DoRA r=32: Approached LoRA r=32 in similarity but skewed to Level 4 (91%), Level 5 nearly absent
• DoRA r=16: Smallest capacity but most equitable priority distribution (L3: 7.7%, L4: 88.5%, L5: 3.9%); best Good+ rate (36.0%) among DoRA variants
• Next step: DoRA r=24 proposed to explore the non-monotone rank–quality tradeoff

Reproduce

# Phase 1: Training (requires .venv-train)
.\.venv-train\Scripts\python.exe train_v09_dora.py --lora-r 16

# Phase 2: Merge (requires .venv-dora)
.\.venv-dora\Scripts\python.exe merge_dora.py

# Phase 3: Inference (requires .venv-dora, NOT .venv-eval)
.\.venv-dora\Scripts\python.exe inference_v082_lora_dora.py --models r16_dora

# Phase 4: SBERT evaluation
.\.venv-eval\Scripts\python.exe evaluate_v08_sbert.py

# Phase 5: Quality Guard + Priority Scoring
.\.venv-eval\Scripts\python.exe pipeline_v08.py

# Phase 6: Figures 35-40 + U.7.2 update
.\.venv-eval\Scripts\python.exe create_u72_dora_ablation.py

# PDF recompile
Set-Location "paper_damage_vlm\1_Methodology"
pdflatex -interaction=nonstopmode methodology_supp2.tex
pdflatex -interaction=nonstopmode methodology_supp2.tex
Set-Location "..\.."

---

🎉 v0.8.5: DoRA r=24 Final Rank Ablation & v1.0.0 Model Selection

Status: ✅ Complete

Completion Date: 2026-06-02
Scope: DoRA r=24 full pipeline; 4-way rank ablation (LoRA r=32 + DoRA r=32/16/24); Figure 15 & Figures 35–40 updated to 4-model comparison; U.7.2 Discussion extended; U.7.3 Qualitative Analysis added; DoRA r=24 selected as v1.0.0 final model

Overview

v0.8.5 completes the DoRA rank ablation study by adding DoRA r=24 — the midpoint proposed in v0.8.4 to resolve the non-monotone rank–quality tradeoff. The experiment confirms that DoRA r=24 achieves the best overall balance among all tested variants, leading to its adoption as the production model for v1.0.0.

DoRA r=24 Training Details

4-Way Ablation Results (n=800)

SBERT Cosine Similarity (2048-dim composite evaluator)

Quality Guard & Priority Distribution (n=800)

Key finding: DoRA r=24 achieves the highest SBERT similarity (mean=0.6791, Good+=37.8%) while maintaining a balanced priority distribution (L3/L4/L5 = 6.0%/84.5%/9.8%). This non-monotone optimum confirms DoRA r=24 as the sweet spot for LLaVA-1.5-7B on this structural damage domain.

Paper Updates

Final Model Selection: DoRA r=24

DoRA r=24 (v0.9) is adopted as the v1.0.0 production model based on:

1. Highest SBERT similarity: mean=0.6791, Good+=37.8% (best across all 7 tested adapters)
2. Balanced priority distribution: L5 rate 9.8% — higher than DoRA r=16 (3.9%) and DoRA r=32 (0.9%), more discriminative than LoRA r=32 (12.8%)
3. Non-collapse output: token σ≈6 (stable), no mode collapse to single level
4. Optimal adapter capacity: 126M trainable parameters — sufficient for bridge-domain fine-grained semantics without over-parameterisation (r=64→collapse; r=16→lower similarity)

v1.0.0 Full Inference (✅ Complete)

Figures Generated

Reproduce

# Phase 1: Training (requires .venv-train)
.\.venv-train\Scripts\python.exe train_v09_dora.py --lora-r 24

# Phase 2: Merge (requires .venv-dora)
.\.venv-dora\Scripts\python.exe merge_dora.py --rank 24

# Phase 3: Inference n=800 (requires .venv-dora)
.\.venv-dora\Scripts\python.exe inference_v082_lora_dora.py --models r24_dora

# Phase 4: SBERT evaluation
.\.venv-eval\Scripts\python.exe evaluate_v08_sbert.py

# Phase 5: Quality Guard + Priority Scoring
.\.venv-eval\Scripts\python.exe pipeline_v08.py

# Phase 6: Figures 35-40 (4-way) + U.7.2 Discussion update
.\.venv-eval\Scripts\python.exe create_u72_dora_4way.py

# Phase 7: U.7.3 Qualitative Analysis
.\.venv-eval\Scripts\python.exe create_u73_qualitative.py

# Phase 8: PDF recompile
Set-Location "paper_damage_vlm\1_Methodology"
pdflatex -interaction=nonstopmode methodology_supp2.tex
pdflatex -interaction=nonstopmode methodology_supp2.tex
Set-Location "..\.."

# Phase 9: v1.0.0 full inference → see v1.0.0 section below
chcp 65001 | Out-Null; [Console]::OutputEncoding = [System.Text.Encoding]::UTF8
.\.venv-dora\Scripts\python.exe -u inference_v10_full.py --checkpoint-every 2000 2>&1 | Tee-Object -FilePath "v10_full_r24_dora_log.txt"

---

🚀 v1.0.0: Production Full Inference (n=10,903)

Status: ✅ Complete — 🎉 Latest Release

Execution period: 2026-06-02 20:20 → 2026-06-03 06:53 JST
Scope: Full-dataset deployment of DoRA r=24 model over all 10,903 records; Quality Guard + Priority Scoring end-to-end; member × damage analysis; Supplementary 2 U.7.4 added to paper

Inference Results

Key finding: PASS rate 97.8% at production scale (n=10,903) is consistent with the validation-set rate. Zero FAIL outputs confirm robust generalisation across the full field dataset. Batch-size-4 processing delivers a 50% speed improvement over single-image inference.

Structural Member & Damage Type Analysis (PASS n=10,668)

Member Frequency

Damage Type Frequency

Key Co-occurrence Findings

Paper Update

Output Files

Reproduce

# Full inference (10,903 rows, batch_size=4, ~10.5 h on RTX 4060 Ti 16GB)
chcp 65001 | Out-Null; [Console]::OutputEncoding = [System.Text.Encoding]::UTF8
.\.venv-dora\Scripts\python.exe -u inference_v10_full.py --checkpoint-every 2000 2>&1 | Tee-Object -FilePath "v10_full_r24_dora_log.txt"

# Member × Damage analysis figure (U.7.4)
.\.venv-eval\Scripts\python.exe create_figures_u74.py

# Recompile paper (run twice for cross-references)
Set-Location "paper_damage_vlm\1_Methodology"
pdflatex -interaction=nonstopmode methodology_supp2.tex
pdflatex -interaction=nonstopmode methodology_supp2.tex
Set-Location "..\.."

---

Performance Metrics

v0.1 Test Results

Priority Distribution (10-Image Test)

• Priority 5 (Immediate Repair Required): 6 images (60%)
• Priority 3 (Planned Maintenance): 4 images (40%)

Resource Utilization

• GPU Usage: 100% (all layers on GPU)
• VRAM: ~8GB / 16GB
• Model Size: 4.08GB (quantized GGUF)
• Processing Speed: ~51.6s/image

---

Setup

1. System Requirements

• OS: Windows 10/11, Linux, or macOS
• GPU: NVIDIA GPU with 8GB+ VRAM (16GB recommended)
• Python: 3.10 or higher
• CUDA: 12.1 or higher
• Storage: 20GB+ free space

2. Clone Repository

git clone https://github.com/your-username/damage_text_score.git
cd damage_text_score

3. Create Virtual Environment

# Windows PowerShell
python -m venv .venv
.venv\Scripts\Activate.ps1

# Linux/macOS
python -m venv .venv
source .venv/bin/activate

4. Install Dependencies

# PyTorch (CUDA 12.4)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

# llama-cpp-python (GPU version)
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124

# Other dependencies
pip install -r requirements.txt

5. Download Models

# LLaVA GGUF Model (Recommended)
python download_llava_gguf.py
# Downloads:
#   - models/ggml-model-q4_k.gguf (4.08GB)
#   - models/mmproj-model-f16.gguf (624MB)

6. Setup Ollama (for JSON Structuring)

# Install Ollama
# https://ollama.com/download

# Pull Swallow-8B model
ollama pull swallow8b-lora-n4000-v09-q4:latest

---

Usage

Quick Start

# Test single image (~42s)
python quickstart.py --mode 1

# Process 10-image batch (~8.5 min)
python quickstart.py --mode 2

# Process 50 images (~43 min)
python quickstart.py --mode 3

# Process all 254 images (~3.6 hours)
python quickstart.py --mode 4

Output Files

data/outputs/
├── quickstart_single.csv        # Single image result
├── quickstart_10images.csv      # 10-image results
├── quickstart_50images.csv      # 50-image results
└── quickstart_254images.csv     # Full dataset results

Output Format

CSV Example:

image_name,damage_type,severity,location,risk,priority_score,priority_level,description
kensg-rebarexposureRb_001.png,crack,high,girder,structural,0.952,5,Extensive cracking observed...

JSON Structure:

{
  "damage_type": "rebar_exposure",
  "severity": "high",
  "location": "girder",
  "risk": "structural",
  "description_ja": "鉄筋露出が見られ、腐食が進行している...",
  "key_features": ["rebar exposure", "moderate corrosion"],
  "priority_score": 0.952,
  "priority_level": 5
}

Custom Usage

from src.pipeline.end_to_end import DamageAnalysisPipeline

# Initialize pipeline
pipeline = DamageAnalysisPipeline("config.yaml")

# Process single image
result = pipeline.process_image("path/to/image.png")

# Batch processing
results = pipeline.process_batch(image_paths, output_csv="results.csv")

---

Model Comparison

Vision Model Performance

Selection Criteria

• llama-cpp-python (Recommended)

  • ✅ Lightweight (4GB)
  • ✅ Full GPU utilization
  • ✅ Ollama-independent
  • ✅ Stable operation
  • ⚠️ Slight accuracy reduction due to quantization

• HuggingFace

  • ✅ Highest accuracy
  • ✅ Full GPU utilization
  • ⚠️ Large size (14GB)
  • ⚠️ High VRAM requirement

• Ollama

  • ⚠️ CPU-only operation (slow)
  • ⚠️ No GPU utilization
  • ✅ Easy setup

---

Tech Stack

Frameworks

• PyTorch 2.6.0 - Deep learning framework
• Transformers 4.57.6 - HuggingFace model hub
• llama-cpp-python 0.3.16 - GGUF inference engine
• OpenCV 4.12.0 - Image processing

Models

• LLaVA-1.5-7B - Vision-Language Model

  • Paper: Visual Instruction Tuning
  • GGUF quantized version (Q4_K_M)

• Swallow-8B - Japanese LLM

  • Developer: TokyoTech LLM Project
  • Specialized for JSON structuring

Libraries

• pandas 2.2.3 - Data manipulation
• pyyaml 6.0.2 - Configuration management
• tqdm 4.67.1 - Progress bars
• pillow 11.1.0 - Image processing

---

Directory Structure

damage_vlm_finetune/
├── .venv-train/                    # Virtual environment: training (v0.7+)
├── .venv-pred/                     # Virtual environment: inference/Unsloth (v0.5+)
├── data/                           # Datasets
│   ├── image_text_inspect_n10789/  # Master dataset (v0.3)
│   │   ├── Rank_c_image_text_n10789.xlsx  # 10,789 annotations
│   │   └── rank_c_images_n10789/   # Image files
│   ├── v03_fine_tuning/            # v0.3 prepared datasets ✅
│   │   ├── test_set_n800.csv       # Fixed test set
│   │   ├── train_1k.csv            # Training: 1k samples
│   │   ├── train_2k.csv            # Training: 2k samples
│   │   ├── train_3k.csv            # Training: 3k samples
│   │   ├── train_4k.csv            # Training: 4k samples
│   │   ├── dataset_preparation_report.md
│   │   ├── evaluations/            # Evaluation results (v0.5)
│   │   └── reports/                # Progressive training reports
│   ├── rebar_exposure_n254/        # v0.2 validation dataset
│   │   └── outputs/                # v0.2 outputs
│   └── preprocessed_640x480_n10789/  # Preprocessed images
├── models/                         # Model files
│   ├── llava-v1.5-7b-Q5_K_M.gguf   # Base model (v0.2)
│   ├── llava_v03_qlora_1k/         # v0.4.1 LoRA adapters ✅
│   ├── llava_v03_qlora_2k/         # v0.4.2 ✅ Recommended
│   ├── llava_v03_qlora_3k/         # v0.4.3 ✅
│   ├── llava_v03_qlora_4k/         # v0.4.4 ✅
│   └── scoring_rules.yaml          # Rule-based scoring
├── src/                            # Source code
│   ├── preprocessing/              # Image preprocessing
│   │   └── image_preprocessor.py
│   ├── vision/                     # Vision analysis (v0.1-v0.2)
│   │   ├── llama_cpp_vision.py     # llama-cpp-python
│   │   ├── gguf_vlm_analyzer.py    # GGUF wrapper
│   │   └── ollama_vision.py        # Ollama integration
│   ├── structuring/                # JSON structuring
│   │   └── json_structurer.py
│   ├── scoring/                    # Priority scoring
│   │   └── priority_scorer.py
│   ├── evaluation/                 # v0.5 evaluation ✅
│   │   ├── __init__.py
│   │   └── vector_similarity_evaluator.py
│   ├── pipeline/                   # Pipeline orchestration
│   │   └── end_to_end.py
│   └── utils/                      # Utilities
│       ├── config.py
│       └── ollama_client.py
├── paper_damage_vlm/               # v0.6 Academic paper
│   ├── 0_Format_arXiv_pdfLaTex/
│   │   ├── bridge_damage_vlm_quantization_2026.tex
│   │   ├── bridge_damage_vlm_quantization_2026.bib
│   │   └── figures/
│   └── 1_Methodology/
├── docs/                           # Documentation
│   ├── ARCHITECTURE.md
│   ├── GPU_SETUP.md
│   ├── VLM_COMPARISON_PLAN_v02.md
│   └── VLM_VALIDATION_REPORT_v02.md
├── prepare_v03_dataset.py          # v0.3 dataset preparation ✅
├── train_v03_qlora.py              # v0.4 QLoRA training ✅
├── create_progressive_training_report.py  # v0.5 reporting ✅
├── config.yaml                     # Unified configuration ✅
├── SETUP_VENV_VLM.md               # v0.3 setup guide ✅
├── QUICKSTART_V03_V05.md           # v0.3-v0.5 workflow ✅
├── V03_V05_IMPLEMENTATION_SUMMARY.md  # Implementation summary ✅
├── requirements.txt                # Python dependencies
├── README.md                       # This file (English)
├── README_JP.md                    # Japanese documentation
├── CHANGELOG.md                    # Version history
└── LICENSE                         # Apache License 2.0

Legend:

• ✅ Implemented: Files/modules completed and tested
• ⏳ Pending: Planned but not yet executed

---

Troubleshooting

Character Encoding Issues (Windows)

Symptom: Japanese characters appear garbled in PowerShell

Solution:

# Change to UTF-8
chcp 65001
python quickstart.py

CUDA Out of Memory

Symptom: CUDA out of memory error

Solution:

# config.yaml
llama_cpp_vision:
  n_gpu_layers: 20  # Reduce from -1 (all layers) to partial GPU

Ollama Connection Error

Symptom: Failed to connect to Ollama

Solution:

# Check Ollama server
ollama list

# Restart server
ollama serve

llama-cpp-python Installation Error

Symptom: Failed building wheel for llama-cpp-python

Solution:

# Install CUDA version explicitly
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124

# Or enable CUDA via environment variable
$env:CMAKE_ARGS="-DLLAMA_CUBLAS=on"
pip install llama-cpp-python --force-reinstall --no-cache-dir

---

Roadmap

✅ Completed

v0.1 (2025-12)

• ✅ End-to-end pipeline implementation
• ✅ Three vision analysis modes (llama-cpp-python, HuggingFace, Ollama)
• ✅ JSON structuring with Swallow-8B
• ✅ Rule-based priority scoring
• ✅ 10-image validation test (100% success rate)

v0.2 (2026-03)

• ✅ LLaVA quantization comparison (Q4_K_M, Q5_K_M, Q8_0)
• ✅ 254-image full dataset processing
• ✅ 5-point quality scoring framework
• ✅ Statistical analysis and visualization
• ✅ Recommendation: Q5_K_M optimal for production

v0.3 (2026-05-22)

• ✅ Dataset preparation pipeline (10,789 images)
• ✅ Quality filtering (text length, keywords, patterns)
• ✅ Stratified sampling (component × damage type)
• ✅ Progressive training sets (1k/2k/3k/4k)
• ✅ Fixed test set (800 images)
• ✅ Comprehensive documentation (setup, quickstart, implementation summary)

v0.4 (2026-05-23)

• ✅ Stage 1: Trained on 1k samples (1:22:57, val_loss=3.135)
• ✅ Stage 2: Trained on 2k samples (2:55:37, val_loss=3.073)
• ✅ Stage 3: Trained on 3k samples (4:31:44, val_loss=3.073)
• ✅ Stage 4: Trained on 4k samples (6:19:10, val_loss=3.067)
• ✅ Generated LoRA adapters for all 4 stages (134.5 MB each)
• ✅ Identified optimal model: 2k samples (best cost-benefit)
• ✅ Comprehensive training analysis report (Result_QLoRA_Scale.md)

v0.5.1 (2026-05-24)

• ✅ Inference pipeline for all 4 fine-tuned models (inference_v051_qlora.py)
• ✅ n=800 evaluation with Japanese Sentence-BERT cosine similarity
• ✅ Inverted-U performance curve: 3k model peaks (mean=0.6909)
• ✅ 4k overfitting confirmed (−2.5% from 3k)
• ✅ 5 publication-quality visualization figures
• ✅ Paper methodology section drafted (v0.5 results)

v0.6.3 (2026-05-25)

• ✅ Two-stage Quality Guard Agent implemented (Rule-based + Swallow-8B SLM)
• ✅ CJK-aware token counting (correct Japanese tokenization)
• ✅ Empirical threshold calibration (θ_low=98, θ_high=202 at 5th/95th percentile)
• ✅ Full n=800 evaluation: PASS=727 (90.9%), FAIL=73 (9.1%)
• ✅ End-to-end throughput: 8.97 s/row (~2.0 h total)
• ✅ Stage 2 (Swallow-8B) saturation finding: 0 additional rejections
• ✅ Output mode collapse discovery: Main Girder 100%, Rebar Exposure 99.6%
• ✅ Priority Level 3 saturation analysis (all 727 PASS → score=0.54)
• ✅ 2 analysis figures generated (violin plots + member/damage heatmap)
• ✅ Paper complete: 25 pages, 10 figures, all sections (pdflatex, no errors)

v0.7.1 (2026-05-26)

• ✅ Image size optimization: confirmed 336px as optimal for LLaVA QLoRA
• ✅ pairdata_v2 construction: noise-removed 2,969-sample corpus (inspect_images_336)
• ✅ S.5 image-size analysis added to methodology_supp.tex

v0.7.2 (2026-05-27)

• ✅ Full progressive QLoRA retraining on pairdata_v2 (1k/2k/3k/4k scales)
• ✅ 3k model selected: peak mean cosine similarity (\u03c1=0.6013), lowest Very Poor rate (21.5%)
• ✅ S.6 semantic similarity evaluation added to methodology_supp.tex
• ✅ 3 evaluation figures generated (similarity by scale, tier distribution, cosine violin)

v0.7.3 (2026-05-28)

• ✅ Quality Guard recalibrated: θ_low 98→30, θ_high 202→200
• ✅ Keyword-based structuring (1s vs 2h Swallow-8B LLM)
• ✅ Full pipeline (n=800): PASS=787 (98.4%), FAIL=13 (1.6%)
• ✅ Scoring saturation resolved: Level 4–5 = 97.3% (was 100% Level 3)
• ✅ Output bias suppression confirmed via member/damage comparison
• ✅ S.7, S.7.1, S.7.2 added to methodology_supp.tex (10 pages)
• ✅ 5 new figures generated (Figure 15–19)

v0.7.4 (2026-05-29)

• ✅ 2048-dim composite embedding evaluator implemented (M1: 1024-dim + M2: 1024-dim)
• ✅ Independent .venv-eval environment (CPU-only, separate from .venv-pred)
• ✅ n=800 evaluation × 4 scales (1k/2k/3k/4k) completed
• ✅ Optimal scale revised: 2k (mean ρ=0.6950, median=0.7053, Good+=53.1%)
• ✅ Very Poor rate dropped from ≤32.9% (384-dim) to ≈0.4% across all scales
• ✅ S.6 fully rewritten with 2048-dim results + mathematical formulas in methodology_supp.tex
• ✅ Figure 12/13/14 regenerated from 2048-dim JSON results
• ✅ SETUP_ENV_EVAL.md created (.venv-eval setup guide)

v0.7.5 (2026-05-29)

• ✅ 2k model established as primary pipeline variant (v0.7.5) based on v0.7.4 optimal-scale finding
• ✅ Figure 19 generated: member & damage type comparison v0.7.3 (3k) vs v0.7.5 (2k)
• ✅ Figure 20 generated: priority score violin for v0.6.3 / v0.7.3 / v0.7.5 (3-version comparison)
• ✅ S.7.1 updated: model scale comparison paragraph (Cross Beam 98.3% in 2k; μ=0.811/std=0.025 vs 3k μ=0.838/std=0.070)
• ✅ S.7.2 updated: Priority Score Distribution violin replaced (old Priority Level bar → new violin figure)
• ✅ S.8 fully updated: 12-panel dual-model qualitative analysis (Pred₁=v0.7.5 2k, Pred₂=v0.7.3 3k)
• ✅ GT/Pred₁ and GT/Pred₂ shared keywords bold-highlighted in all 12 panels
• ✅ Image size reduced to 0.675\linewidth (×0.9 from 0.75) in all S.8 figures
• ✅ methodology_supp.pdf: 17 pages, no errors (pdflatex)

v0.8 (2026-05-30)

• ✅ pairdata_v3 denoising pipeline: 10,789 → 2,500-sample training corpus
• ✅ LoRA rank ablation on pairdata_v3 (r=16/32/64, 2,500 samples each)
  • r=16: eval_loss=3.138, token_acc=98.73%, 3h24m
  • r=32: eval_loss=3.133, token_acc=98.83%, 5h36m
  • r=64: eval_loss=3.122, token_acc=98.90%, 3h27m (selected)
• ✅ r=64 selected as optimal (168.8M params / 4.56% of backbone, no extra GPU cost)
• ✅ Supplementary 2 (methodology_supp2.tex) fully documented: U.1–U.8
• ✅ Figure 21: LoRA rank ablation validation loss trajectories (pairdata_v3)
• ✅ U.4 Analysis: adapter capacity argument (168M vs 42M/84M for domain specificity)

v0.8.1 (2026-05-30)

• ✅ DoRA training (use_dora=True, r=64, alpha=128) on pairdata_v3 2,500 samples
• ✅ DoRA eval_loss: 3.119 (best, epoch 2.421) — −0.003 improvement over LoRA r=64 (3.122)
• ✅ DoRA token_acc: 98.92% (best), 98.95% (final), train_loss=3.417, 6h51m
• ✅ Research question (i) validated: DoRA achieves lower eval_loss at same rank/VRAM
• ✅ U.8 in Supplementary 2 fully updated: Config (r=32→r=64), Results table (TBD→actual), Repro Commands
• ✅ U.8 Motivation rewritten: r=64 basis, 3 research questions (val loss, cosine sim, hallucination)
• ✅ methodology_supp2.pdf: 15 pages, no errors (pdflatex)

v0.8.2 (2026-05-31)

• ✅ 2048-dim Sentence-BERT evaluation on all 4 adapters (n=800 test set)
• ✅ DoRA r=64: highest embedding score (cosine 0.6846, Good+ 41.0%) but output-level mode collapse confirmed
• ✅ Quality Guard + Priority Scoring: both models PASS 99.8%; DoRA all Level 4 (100%), LoRA r=32 L3/4/5 distributed
• ✅ U.6.1 Structural Member & Damage Type analysis: DoRA stereotyped (Main Girder+Deck Slab 100%, 3 damage types 100%)
• ✅ DoRA r=64 fine-tuning declared failed — r=64 likely excessive; DoRA protocol left as future work
• ✅ LoRA r=32 (v0.8) adopted as final pipeline model
• ✅ Supplementary 2 updated: U.5 table+figures+analysis, U.6 2-model comparison, U.6.1 new subsection (Figure 28)
• ✅ methodology_supp2.pdf: 21 pages

v0.8.3 (2026-05-31)

• ✅ U.7 qualitative analysis section fully rebuilt on pairdata v3 (inference_r32.csv)
• ✅ 12 representative PASS images re-selected with greedy member × damage diversity
• ✅ Caption simplified to 3-item format: Quality Metrics / GT / Pred
• ✅ Shared keyword highlighting added (CJK gothic font)
• ✅ s8 folder replaced (figures_vlm/s8/s8_01–s8_12)
• ✅ methodology_supp2.pdf: 19 pages, no errors

v0.8.4 (2026-06-01)

• ✅ DoRA r=16 full pipeline: train (eval_loss=3.138, 6h48m) → merge (85 shards) → inference (800 rows, 798 OK) → SBERT (mean=0.6717, Good+=36.0%) → pipeline (PASS=99.6%, L3:61/L4:705/L5:31)
• ✅ 3-way rank ablation confirmed: LoRA r=32 vs. DoRA r=32 vs. DoRA r=16
• ✅ Key finding: DoRA r=64 mode collapse (all Level 4); DoRA r=16 most balanced priority distribution
• ✅ Figure 41: validation loss trajectories for 3-way ablation (added to U.7.1)
• ✅ Figures 35–40: 6-panel 3-way comparison (generated by create_u72_dora_ablation.py)
• ✅ U.7.2 replaced: title → “DoRA Rank Ablation Studies: LoRA r=32 vs. DoRA r=32/16”
• ✅ U.7.2 Discussion added: gradient collapse analysis, r=16 equitable output, DoRA r=24 proposed
• ✅ methodology_supp2.pdf: 5648 KB, no errors

v0.8.5 (2026-06-02)

• ✅ DoRA r=24 full pipeline: train (eval_loss=3.137, ~6h50m) → merge → inference (800 rows, 798 OK)
• ✅ SBERT: mean=0.6791, std=0.0635, median=0.6834, Good+=37.8% — best among all 7 adapters tested
• ✅ Quality Guard + Priority: PASS=798 (99.8%), L3=48/L4=674/L5=78, token_mean=90.6
• ✅ DoRA r=24 selected as v1.0.0 final model (peak SBERT + balanced L3/4/5 distribution)
• ✅ Figure 15 updated: “four models”; DoRA r=24 (eval_loss=3.137) added as 4th validation loss curve
• ✅ Figures 35–40: regenerated as 4-way comparison (LoRA r=32, DoRA r=32/16/24)
• ✅ U.7.2 Discussion extended to 6 paragraphs: rank–quality relationship + figure-level evidence + final model selection
• ✅ U.7.3 added: Qualitative Analysis: Finetuned Model Predictions DoRA r=24 (12 panels)
• ✅ methodology_supp2.pdf: rebuilt, no errors

🔮 Future Work

v1.0.0 (2026-06-03) 🎉 Latest Release — Full Dataset Inference

• ✅ Full inference completed: 10,903 rows, DoRA r=24, batch_size=4, 10.55 h (3.5 s/image)
• ✅ Quality Guard PASS=10,668 (97.8%), FAIL=0 (0.0%), MISSING=235 (2.2%)
• ✅ Priority: L3=454 (4.3%) / L4=9,185 (86.1%) / L5=1,029 (9.6%)
• ✅ Output: data/v10_full/v10_full_r24_dora.xlsx (10,903 rows)
• ✅ Member x damage figure: create_figures_u74.py -> figures_vlm/u74_member_damage_analysis.png
• ✅ U.7.4 added to Supplementary 2: inference config, results table, figure, discussion
• ✅ methodology_supp2.pdf: rebuilt, no errors

v1.1 (2026 Q3) - Production Deployment

• Web UI (Streamlit/Gradio)
• REST API server (FastAPI)
• Docker containerization
• CI/CD pipeline
• Comprehensive unit tests
• Performance benchmarking

Research Extensions

• Balanced training corpus to address output mode collapse
• Constrained-generation prompts (per-member damage reporting)
• Priority scoring recalibration with explicit severity keywords
• Multi-damage type support (crack detection, leakage, scaling)
• Active learning for annotation efficiency
• Multi-language support (Japanese ↔ English)
• Real-time video analysis
• Integration with bridge inspection databases
• Explainable AI (attention visualization, saliency maps)
• Alternative VLMs (Qwen2-VL, InternVL2, GPT-4V)

---

Citation

If you use this project in your research, please cite:

@software{bridge_damage_assessment_2026,
  title = {Bridge Damage Assessment and Repair Priority Scoring System},
  author = {Your Name},
  year = {2026},
  version = {0.1.0},
  url = {https://github.com/your-username/damage_text_score}
}

---

References

1. Liu et al. (2023). "Visual Instruction Tuning" - LLaVA [arXiv:2304.08485]
2. TokyoTech LLM Project - Swallow Models [GitHub]
3. Georgi Gerganov - llama.cpp [GitHub]

---

License

Apache License 2.0 - See LICENSE for details

---

---

📈 v0.7.3: Quality Guard Recalibration & Full Pipeline

Status: ✅ Complete

Completion Date: 2026-05-28
Scope: Quality Guard recalibration + keyword structuring pipeline on 3k pairdata_v2 model (n=800)

What Was Completed

• Quality Guard recalibration: θ_low 98 → 30 (captures only ERROR outputs); θ_high 202 → 200
  • Old threshold rejected all valid short descriptions produced by the pairdata_v2 model
  • New threshold retains all structurally valid predictions
• Keyword-based structuring: Replaced 2-hour Swallow-8B LLM with 1-second keyword matching
• Full pipeline run (pipeline_v073.py): n=800 in < 2 seconds
• Output mode collapse resolved: pairdata_v2 noise removal suppressed hallucinated co-occurrence patterns

Pipeline Results (n=800)

Priority Level Distribution (PASS n=787)

Contrast with v0.6.3 where 100% of PASS samples scored Level 3 (scoring saturation at 0.54).

Key Findings

• Scoring saturation resolved: Level 4–5 now accounts for 97.3% of PASS predictions
• Bias suppression: Lime Efflorescence dropped 49.0% → 13.3%; Curb emerged as secondary member (92.5%)
• PASS rate: 98.4% vs. 90.9% in v0.6.3 (quality improvement from pairdata_v2 denoising)
• Throughput: Full 800-image pipeline in < 2 seconds

Figures Generated

Repro Commands

.\.venv-pred\Scripts\Activate.ps1
python pipeline_v073.py

Paper Artifacts Updated

• methodology_supp.tex — S.7 Quality Guard Recalibration
• methodology_supp.tex — S.7.1 Member & Damage Type comparison
• methodology_supp.tex — S.7.2 Priority Level comparison
• methodology_supp.pdf — 10 pages, no errors

---

Last Updated: May 29, 2026 (v0.7.3 — Quality Guard recalibration, full pipeline, scoring saturation resolved)

---

🔢 v0.7.4: 2048-Dimensional Composite Embedding Evaluation

Status: ✅ Complete — 🎉 Latest Release

Completion Date: 2026-05-29
Scope: Extended semantic similarity evaluation using a 2,048-dimensional composite embedding (n=800 test set × 4 scales)

Overview

v0.7.4 replaces the 384-dimensional single-model evaluator (paraphrase-multilingual-MiniLM-L12-v2) with a 2,048-dimensional composite embedding formed by combining two complementary Japanese-capable sentence encoders. This yields substantially higher and more discriminative cosine similarity scores and revises the optimal training scale from 3k to 2k.

Composite Evaluator Architecture

Mathematical Formulation

Given ground-truth text $t^{gt}$ and prediction $t^{pred}$:

1. Each encoder produces an $L_2$-normalised embedding:
   • $\mathbf{e}1(t) = f{M1}(t) / |f_{M1}(t)|$
   • $\mathbf{e}2(t) = f{M2}(\text{"query: "} | t) / |f_{M2}(\cdot)|$
2. Concatenate and re-normalise (each half has unit norm, so denominator is $\sqrt{2}$):
   • $\mathbf{e}(t) = [\mathbf{e}_1; \mathbf{e}_2] / \sqrt{2}$
3. Cosine similarity reduces to the arithmetic mean of the two per-model similarities:
   • $\rho_{2048} = \mathbf{e}^{gt} \cdot \mathbf{e}^{pred} = (\rho_1 + \rho_2) / 2$

Evaluation Results (2048-dim)

Test set: n=800 (pairdata-v2 ground truth), CPU inference
Tier thresholds: Excellent ≥ 0.85 · Good ≥ 0.70 · Acceptable ≥ 0.55 · Poor ≥ 0.40 · Very Poor < 0.40

Comparison: 384-dim vs 2048-dim Evaluators

Very Poor rate: 384-dim was 21.5–32.9%; 2048-dim is uniformly ≈0.4%.

Key Findings

1. Optimal scale revised to 2k: 2k achieves mean ρ=0.6950 and median=0.7053 (above Good threshold), with 53.1% Good+ — significantly better than 3k/4k.
2. Overfitting pattern confirmed: 2k peaks, then 3k/4k both regress, consistent with pairdata-v2 diminishing-returns pattern found in token-accuracy analysis.
3. Very Poor near-elimination: 2048-dim composite scores are far more discriminative; catastrophic mismatch cases detected by 384-dim (VPoor ≤32.9%) essentially disappear (≈0.4%).
4. Good+ majority at 2k: 53.1% of 2k predictions reach the Good+ tier, reflecting M2’s stronger multilingual semantic understanding.

Figures Updated

Paper Artifacts Updated

• paper_damage_vlm/1_Methodology/methodology_supp.tex — S.6 fully rewritten: evaluator description, Table with 2048-dim values, Key findings (2k peak), Figure 12/13/14 captions, Model selection (3k → 2k), Extended Evaluation subsubsection with three mathematical formulas
• paper_damage_vlm/1_Methodology/methodology_supp.pdf — 14 pages, recompiled (no errors)

New Files

Repro Commands

# 1. Create evaluation environment (CPU-only, independent of .venv-pred)
python -m venv .venv-eval
.venv-eval\Scripts\pip.exe install "torch==2.6.0+cpu" --index-url https://download.pytorch.org/whl/cpu
.venv-eval\Scripts\pip.exe install "sentence-transformers==3.3.1" numpy pandas scikit-learn matplotlib seaborn

# 2. Run evaluation (n=800 x 4 scales, ~26 min total on CPU)
.venv-eval\Scripts\python.exe evaluate_2048dim.py

# 3. Regenerate Figure 12/13/14
.venv-eval\Scripts\python.exe create_figures_v072_2048dim.py

Output JSON files: data/v07_fine_tuning/evaluations_2048dim/evaluation_2048dim_{1k,2k,3k,4k}.json

---

📊 v0.7.5: 2k Pipeline Evaluation & Supplementary Paper Update

Status: ✅ Complete

Last Updated: May 29, 2026
Scope: 2k model established as primary pipeline; cross-model comparison figures; full S.7–S.8 supplementary paper update

Overview

Building on the v0.7.4 finding that the 2k model is optimal (mean ρ=0.6950, Good+=53.1%), v0.7.5 designates the 2k QLoRA pipeline as the primary production variant and adds cross-model comparative analysis to the supplementary paper.

Key Deliverables

Figure 19: Member & Damage Type Comparison (v0.7.3 vs v0.7.5)

figures_vlm/19_member_damage_v073_v075.png

Side-by-side frequency bars and co-occurrence heatmaps comparing the 3k model (v0.7.3) against the 2k model (v0.7.5).

Notable finding — Cross Beam saturation in 2k:

The 2k model systematically appends cross beam descriptors in 98.3% of all predictions, reflecting template-driven vocabulary learned from the smaller training corpus.

Figure 20: Priority Score Violin — 3-Version Comparison

figures_vlm/20_priority_score_violin.png

Violin plots comparing priority score distributions across all three pipeline versions:

Interpretation: v0.6.3 collapses to a single value (mode-collapsed outputs). Both v0.7.x models restore discriminability; v0.7.5 (2k) shows a tighter, more predictable Level-4 cluster (std=0.025) while v0.7.3 (3k) reaches higher scores with greater variance (std=0.070), reflecting its longer training.

S.8 Qualitative Analysis — Dual-Model Comparison (12 Panels)

paper_damage_vlm/1_Methodology/methodology_supp.tex — Section S.8

All 12 representative PASS panels updated to display three rows per image:

Additional formatting changes applied to all 12 panels:

• Sim(emb) values reflect the 2k model (Pred₁)
• Image size reduced to 0.675\linewidth (×0.9 from 0.75)
• Caption: "GT/Pred$_1$ shared terms bold in Pred$_1$; GT/Pred$_2$ shared terms bold in Pred$_2$"

Cosine Similarity (Sim(emb)) — 12 Selected Panels

The 2k model achieves higher Sim(emb) in 9 of 12 panels; the 3k model leads in 3 (panels 03, 07, 12).

Scripts

Paper Artifacts Updated

• paper_damage_vlm/1_Methodology/methodology_supp.tex — S.7.1, S.7.2, S.8 fully updated
• paper_damage_vlm/1_Methodology/methodology_supp.pdf — 17 pages, no errors (pdflatex)

Repro Commands

# Generate Figure 19 and Figure 20
.venv-eval\Scripts\python.exe create_figures_v075_s71_s72.py

# Recompile supplementary PDF
cd paper_damage_vlm\1_Methodology
pdflatex -interaction=nonstopmode methodology_supp.tex

---

Last Updated: June 1, 2026 (v0.8.4 — DoRA rank ablation r=16/32 full pipeline complete; DoRA r=16 achieves best Good+ rate 36.0% and most balanced priority distribution; U.7.2 updated with 3-way comparison Figures 35–41; DoRA r=24 proposed as next experiment)