An end-to-end pipeline for automated analysis of bridge structural damage (rebar exposure, cracks, corrosion) using **LLaVA (Large Language and Vision Assistant)**. The system generates expert-level damage descriptions from images and produces structured prioritization scores for repair planning.
Python
1
6 commits
updated Mar 31, 2026
Automated Bridge Damage Analysis and Repair Prioritization using Vision-Language Models (LLaVA)
π ζ₯ζ¬θͺηγγγ₯γ‘γ³γ (Japanese Documentation)
An end-to-end pipeline for automated analysis of bridge structural damage (rebar exposure, cracks, corrosion) using LLaVA (Large Language and Vision Assistant). The system generates expert-level damage descriptions from images and produces structured prioritization scores for repair planning.
graph TB
A[Input Images<br/>254 Bridge Damage Photos] --> B[Preprocessing<br/>Denoise/Resize/Contrast]
B --> C[Vision Analysis<br/>LLaVA-1.5-7B GGUF]
C --> D[JSON Structuring<br/>Swallow-8B]
D --> E[Priority Scoring<br/>Rule-Based Engine]
E --> F[Output<br/>CSV/JSON Reports]
style A fill:#e1f5ff
style C fill:#fff4e1
style E fill:#ffe1e1
style F fill:#e1ffe1
flowchart LR
subgraph Input["π Input Stage"]
A1[Raw Images<br/>PNG/JPG]
end
subgraph Preprocess["π§ Preprocessing"]
B1[Noise Reduction<br/>NL-Means]
B2[Resize<br/>Max 1024x1024]
B3[Contrast<br/>CLAHE]
end
subgraph Vision["ποΈ Vision Analysis"]
C1[Image Encoding<br/>CLIP Vision]
C2[LLaVA Inference<br/>Q4_K_M Quantized]
C3[Damage Description<br/>Japanese Text]
end
subgraph Structure["π Structuring"]
D1[Swallow-8B LLM<br/>via Ollama]
D2[JSON Parsing<br/>Multi-Pattern]
D3[Structured Data<br/>Type/Severity/Location]
end
subgraph Score["π― Scoring"]
E1[Rule Evaluation<br/>40% Severity<br/>35% Type<br/>15% Location<br/>10% Risk]
E2[Priority Calculation<br/>0.0-1.0 Score]
E3[Level Assignment<br/>1-5 Priority]
end
subgraph Output["πΎ Output"]
F1[CSV Export<br/>Batch Results]
F2[JSON Export<br/>Detailed Records]
end
A1 --> B1 --> B2 --> B3
B3 --> C1 --> C2 --> C3
C3 --> D1 --> D2 --> D3
D3 --> E1 --> E2 --> E3
E3 --> F1
E3 --> F2
style Input fill:#e1f5ff
style Vision fill:#fff4e1
style Score fill:#ffe1e1
style Output fill:#e1ffe1
| Stage | Module | Purpose | Technology |
|---|---|---|---|
| Preprocessing | image_preprocessor.py | Image quality enhancement | OpenCV 4.12 |
| Vision Analysis | llama_cpp_vision.py | Damage description generation | LLaVA-1.5-7B Q4_K_M (4GB) |
| Structuring | json_structurer.py | Natural language β JSON | Swallow-8B (Ollama) |
| Scoring | priority_scorer.py | Repair priority calculation | Rule-based (YAML) |
| Pipeline | end_to_end.py | Orchestration | Python 3.12 |
Processing Time Breakdown (per image):
βββββββββββββββββββββββββββββββββββββββββββ
β Preprocessing: ~2s (4%) β
β Vision Analysis: ~42s (81%) β
β JSON Structuring: ~5s (10%) β
β Scoring: <1s (2%) β
βββββββββββββββββββββββββββββββββββββββββββ€
β Total: ~51.6 seconds/image β
βββββββββββββββββββββββββββββββββββββββββββ
3 Vision Modes Implemented
Complete Pipeline
Validation Tests
Windows Encoding Issues Resolved
Comprehensive evaluation of LLaVA-1.5-7B quantization levels (Q4_K_M, Q5_K_M, Q8_0) to determine the optimal balance between accuracy, speed, and model size for bridge damage assessment.
Developed a 5-point quality scoring system (v0.2 design) to assess damage description completeness:
| Component | Max Points | Criteria |
|---|---|---|
| Damage Types | 2.0 | Recognition of crack, rebar exposure, corrosion, spalling, section loss |
| Severity Level | 1.0 | Minor, moderate, severe classification |
| Location Info | 1.0 | Spatial information (top, bottom, left, right, etc.) |
| Extent Info | 1.0 | Coverage description (local, widespread, partial, etc.) |
| Total | 5.0 | Comprehensive damage assessment score |
| Quantization | Model Size | Init Time | Avg Inference | Quality Score | Text Length | Success Rate |
|---|---|---|---|---|---|---|
| Q4_K_M | 4.1GB | 3.6s | 5.43s | 2.93 Β± 1.53 | 168 Β± 41 | 254/254 (100%) |
| Q5_K_M β | 4.8GB | 4.5s | 5.67s | 3.18 Β± 1.35 | 160 Β± 37 | 254/254 (100%) |
| Q8_0 | 7.2GB | 5.9s | 7.63s | 3.27 Β± 1.39 | 162 Β± 39 | 254/254 (100%) |
Performance Comparison:

Figure 1: Performance metrics across quantization levels (N=254). Shows average inference time, model size vs total time, quality score comparison, and summary statistics.
π English Version | Japanese Version

Figure 2: Comprehensive statistical analysis including violin plots (quality, text length, inference time), box plots, scatter plots (quality vs text length/inference time), and detailed statistics table.
Violin Plot Analysis:
Statistical Significance (Mann-Whitney U Test):
| Quantization | Correlation Coefficient | Interpretation |
|---|---|---|
| Q4_K_M | -0.559 | Moderate negative (longer β lower quality) |
| Q5_K_M | -0.148 | Weak negative (stable across lengths) |
| Q8_0 | -0.393 | Moderate negative |
Insight: Q5_K_M maintains consistent quality regardless of description length, indicating robust performance.
Speed-Quality Efficiency:
Q4_K_M: 0.54 quality/sec (2.93 / 5.43)
Q5_K_M: 0.56 quality/sec (3.18 / 5.67) β BEST β
Q8_0: 0.43 quality/sec (3.27 / 7.63)
Best Quality-Speed Balance
Stable Performance
Resource Efficiency
| Priority | Quantization | Use Case |
|---|---|---|
| π₯ Recommended | Q5_K_M | Production deployment (best balance) |
| π₯ Alternative | Q8_0 | High-accuracy applications (if speed is not critical) |
| π₯ Development | Q4_K_M | Fast prototyping only |
For Bridge Damage Assessment:
Quantization is Not Free
Sweet Spot Exists
Statistical Validation is Essential
GPU Compatibility Matters
Quality Metrics Enable Optimization
Domain-specific fine-tuning of LLaVA-1.5-7B (Q5_K_M) using QLoRA (Quantized Low-Rank Adaptation) to improve damage description quality from 3.18/5.0 β 4.0+/5.0 while maintaining inference speed.
Based on v0.2 results:
Q4_K_M rejected due to bimodal quality distribution (high variance). Q8_0 rejected due to poor cost-benefit ratio.
Base Model: LLaVA-1.5-7B (Q5_K_M quantization)
Adapter: LoRA (Low-Rank Adaptation)
Target Modules:
- q_proj, v_proj (attention layers)
- vision_tower (optional, for domain-specific visual features)
LoRA Rank: 16-32 (balance between capacity and efficiency)
LoRA Alpha: 32-64 (scaling factor)
Dropout: 0.05-0.1 (prevent overfitting)
Quantization: 4-bit (during training, base model frozen)
Source: 254 rebar exposure images + human-annotated ground truth
Data Preparation:
High-Quality Subset Selection (N=50-100):
Annotation Format:
{
"image": "kensg-rebarexposureRb_001.png",
"ground_truth": "Severe rebar exposure with extensive corrosion covering approximately 60% of the bottom-right beam section. Multiple horizontal cracks (3-5mm width) extend across the surface. Significant concrete spalling reveals rebar with heavy rust accumulation. Section loss estimated at 15-20mm depth.",
"quality_score": 5.0,
"damage_types": ["rebar_exposure", "corrosion", "crack", "spalling", "section_loss"],
"severity": "severe",
"location": "bottom_right_beam",
"extent": "60%"
}
Data Augmentation:
| Parameter | Value | Rationale |
|---|---|---|
| Batch Size | 4-8 | Fit in 16GB VRAM (gradient accumulation if needed) |
| Learning Rate | 1e-4 to 5e-4 | Typical for LoRA fine-tuning |
| Epochs | 3-5 | Avoid overfitting on small dataset |
| Optimizer | AdamW | Standard for transformers |
| LR Schedule | Cosine with warmup | Smooth convergence |
| Gradient Clipping | 1.0 | Stability |
| Mixed Precision | FP16 | Memory efficiency |
graph LR
A[254 Images] --> B[Manual Annotation<br/>50-100 samples]
B --> C[Train/Val Split<br/>80/20]
C --> D[QLoRA Training<br/>Q5_K_M base]
D --> E[Adapter Weights<br/>~10-50MB]
E --> F[Merge with Base<br/>Optional]
F --> G[Validation<br/>254 images]
G --> H{Quality > 4.0?}
H -->|Yes| I[Deploy v0.3]
H -->|No| J[Hyperparameter Tuning]
J --> D
| Metric | v0.2 (Q5_K_M) | v0.3 (Fine-Tuned) | Improvement |
|---|---|---|---|
| Quality Score | 3.18 Β± 1.35 | 4.0-4.5 Β± 0.8 | +26-41% |
| Damage Type Coverage | 2.93/4 keywords | 3.5-4.0/4 | +19-37% |
| Severity Accuracy | 65% | 85-90% | +31-38% |
| Location Precision | 58% | 80-85% | +38-47% |
| Extent Quantification | 42% | 70-75% | +67-79% |
| Inference Speed | 5.67s | 5.5-6.0s | ~0% (maintained) |
β Minimum Acceptable Performance:
π― Target Performance:
peft, bitsandbytes, transformers)# QLoRA fine-tuning
pip install peft>=0.11.0 # Parameter-Efficient Fine-Tuning
pip install bitsandbytes>=0.43.0 # 4-bit quantization
pip install transformers>=4.41.0 # HuggingFace transformers
pip install accelerate>=0.30.0 # Distributed training utilities
# Training monitoring
pip install wandb # Experiment tracking (optional)
pip install tensorboard # Local logging (optional)
# Data preparation
pip install datasets # HuggingFace datasets
Option A: Fine-tune FP16 model β Convert to GGUF
llama.cpp toolsOption B: Use unsloth (optimized QLoRA)
unsloth library: 2x faster training, 50% less VRAMOption C: Cloud training (if local VRAM insufficient)
| Risk | Mitigation Strategy |
|---|---|
| Overfitting (small dataset N=50-100) | Strong regularization (dropout 0.1), early stopping, data augmentation |
| Catastrophic forgetting | Low learning rate (1e-4), small LoRA rank (16-32), freeze base model |
| Inference slowdown | Profile adapter overhead, consider merging weights, quantize adapter |
| Annotation quality | Inter-rater agreement (Cohen's kappa >0.7), expert review |
| GGUF compatibility | Start with HF model, convert after training, validate outputs |
| Test Scale | Processing Time | Success Rate | Avg Time/Image |
|---|---|---|---|
| Single Image | 42s | 100% | 42s |
| 10-Image Batch | 8m 35s | 100% | 51.6s |
| 50-Image (Est.) | ~43m | - | ~52s |
| 254-Image (Est.) | ~3.6h | - | ~51s |
git clone https://github.com/your-username/damage_text_score.git
cd damage_text_score
# Windows PowerShell
python -m venv .venv
.venv\Scripts\Activate.ps1
# Linux/macOS
python -m venv .venv
source .venv/bin/activate
# PyTorch (CUDA 12.4)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
# llama-cpp-python (GPU version)
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124
# Other dependencies
pip install -r requirements.txt
# LLaVA GGUF Model (Recommended)
python download_llava_gguf.py
# Downloads:
# - models/ggml-model-q4_k.gguf (4.08GB)
# - models/mmproj-model-f16.gguf (624MB)
# Install Ollama
# https://ollama.com/download
# Pull Swallow-8B model
ollama pull swallow8b-lora-n4000-v09-q4:latest
# Test single image (~42s)
python quickstart.py --mode 1
# Process 10-image batch (~8.5 min)
python quickstart.py --mode 2
# Process 50 images (~43 min)
python quickstart.py --mode 3
# Process all 254 images (~3.6 hours)
python quickstart.py --mode 4
data/outputs/
βββ quickstart_single.csv # Single image result
βββ quickstart_10images.csv # 10-image results
βββ quickstart_50images.csv # 50-image results
βββ quickstart_254images.csv # Full dataset results
CSV Example:
image_name,damage_type,severity,location,risk,priority_score,priority_level,description
kensg-rebarexposureRb_001.png,crack,high,girder,structural,0.952,5,Extensive cracking observed...
JSON Structure:
{
"damage_type": "rebar_exposure",
"severity": "high",
"location": "girder",
"risk": "structural",
"description_ja": "ιηι²εΊγθ¦γγγθ
ι£γι²θ‘γγ¦γγ...",
"key_features": ["rebar exposure", "moderate corrosion"],
"priority_score": 0.952,
"priority_level": 5
}
from src.pipeline.end_to_end import DamageAnalysisPipeline
# Initialize pipeline
pipeline = DamageAnalysisPipeline("config.yaml")
# Process single image
result = pipeline.process_image("path/to/image.png")
# Batch processing
results = pipeline.process_batch(image_paths, output_csv="results.csv")
| Mode | Model | Size | Time/Image | GPU Usage | Rating |
|---|---|---|---|---|---|
| llama-cpp-python | LLaVA-1.5-7B Q4_K_M | 4.08GB | 51.6s | 100% | βββββ |
| HuggingFace | llava-1.5-7b-hf | 14GB | 45s | 100% | ββββ |
| Ollama | llava:7b | 4.7GB | 88s | 0% (CPU) | ββ |
llama-cpp-python (Recommended)
HuggingFace
Ollama
LLaVA-1.5-7B - Vision-Language Model
Swallow-8B - Japanese LLM
damage_text_score/
βββ .venv/ # Python virtual environment
βββ data/ # Dataset
β βββ images_human_inspect_n254/ # Input images (254 files)
β βββ preprocessed/ # Preprocessed images
β βββ outputs/ # Processing results
β βββ descriptions/ # Vision outputs
β βββ structured/ # JSON structured outputs
β βββ scores/ # Scoring results
βββ models/ # Model files
β βββ ggml-model-q4_k.gguf # LLaVA GGUF (4.08GB)
β βββ mmproj-model-f16.gguf # MMProj (624MB)
β βββ scoring_rules.yaml # Scoring rules
βββ src/ # Source code
β βββ preprocessing/ # Preprocessing module
β β βββ image_preprocessor.py
β βββ vision/ # Vision analysis
β β βββ llama_cpp_vision.py # llama-cpp-python (Recommended)
β β βββ granite_vision.py # HuggingFace version
β β βββ ollama_vision.py # Ollama version
β βββ structuring/ # JSON structuring
β β βββ json_structurer.py
β βββ scoring/ # Scoring
β β βββ priority_scorer.py
β βββ pipeline/ # Pipeline orchestration
β β βββ end_to_end.py
β βββ utils/ # Utilities
β βββ config.py
β βββ ollama_client.py
βββ config.yaml # System configuration
βββ quickstart.py # Quick start script
βββ download_llava_gguf.py # Model download script
βββ requirements.txt # Python dependencies
βββ README.md # This file (English)
βββ README_JP.md # Japanese documentation
βββ CHANGELOG.md # Version history
βββ LICENSE # MIT License
Symptom: Japanese characters appear garbled in PowerShell
Solution:
# Change to UTF-8
chcp 65001
python quickstart.py
Symptom: CUDA out of memory error
Solution:
# config.yaml
llama_cpp_vision:
n_gpu_layers: 20 # Reduce from -1 (all layers) to partial GPU
Symptom: Failed to connect to Ollama
Solution:
# Check Ollama server
ollama list
# Restart server
ollama serve
Symptom: Failed building wheel for llama-cpp-python
Solution:
# Install CUDA version explicitly
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124
# Or enable CUDA via environment variable
$env:CMAKE_ARGS="-DLLAMA_CUBLAS=on"
pip install llama-cpp-python --force-reinstall --no-cache-dir
If you use this project in your research, please cite:
@software{bridge_damage_assessment_2026,
title = {Bridge Damage Assessment and Repair Priority Scoring System},
author = {Your Name},
year = {2026},
version = {0.1.0},
url = {https://github.com/your-username/damage_text_score}
}
MIT License - See LICENSE for details
Last Updated: March 20, 2026 (v0.1.0)
An end-to-end pipeline for automated analysis of bridge structural damage (rebar exposure, cracks, corrosion) using **LLaVA (Large Language and Vision Assistant)**. The system generates expert-level damage descriptions from images and produces structured prioritization scores for repair planning.
Python
1
6 commits
updated Mar 31, 2026
Automated Bridge Damage Analysis and Repair Prioritization using Vision-Language Models (LLaVA)
π ζ₯ζ¬θͺηγγγ₯γ‘γ³γ (Japanese Documentation)
An end-to-end pipeline for automated analysis of bridge structural damage (rebar exposure, cracks, corrosion) using LLaVA (Large Language and Vision Assistant). The system generates expert-level damage descriptions from images and produces structured prioritization scores for repair planning.
graph TB
A[Input Images<br/>254 Bridge Damage Photos] --> B[Preprocessing<br/>Denoise/Resize/Contrast]
B --> C[Vision Analysis<br/>LLaVA-1.5-7B GGUF]
C --> D[JSON Structuring<br/>Swallow-8B]
D --> E[Priority Scoring<br/>Rule-Based Engine]
E --> F[Output<br/>CSV/JSON Reports]
style A fill:#e1f5ff
style C fill:#fff4e1
style E fill:#ffe1e1
style F fill:#e1ffe1
flowchart LR
subgraph Input["π Input Stage"]
A1[Raw Images<br/>PNG/JPG]
end
subgraph Preprocess["π§ Preprocessing"]
B1[Noise Reduction<br/>NL-Means]
B2[Resize<br/>Max 1024x1024]
B3[Contrast<br/>CLAHE]
end
subgraph Vision["ποΈ Vision Analysis"]
C1[Image Encoding<br/>CLIP Vision]
C2[LLaVA Inference<br/>Q4_K_M Quantized]
C3[Damage Description<br/>Japanese Text]
end
subgraph Structure["π Structuring"]
D1[Swallow-8B LLM<br/>via Ollama]
D2[JSON Parsing<br/>Multi-Pattern]
D3[Structured Data<br/>Type/Severity/Location]
end
subgraph Score["π― Scoring"]
E1[Rule Evaluation<br/>40% Severity<br/>35% Type<br/>15% Location<br/>10% Risk]
E2[Priority Calculation<br/>0.0-1.0 Score]
E3[Level Assignment<br/>1-5 Priority]
end
subgraph Output["πΎ Output"]
F1[CSV Export<br/>Batch Results]
F2[JSON Export<br/>Detailed Records]
end
A1 --> B1 --> B2 --> B3
B3 --> C1 --> C2 --> C3
C3 --> D1 --> D2 --> D3
D3 --> E1 --> E2 --> E3
E3 --> F1
E3 --> F2
style Input fill:#e1f5ff
style Vision fill:#fff4e1
style Score fill:#ffe1e1
style Output fill:#e1ffe1
| Stage | Module | Purpose | Technology |
|---|---|---|---|
| Preprocessing | image_preprocessor.py | Image quality enhancement | OpenCV 4.12 |
| Vision Analysis | llama_cpp_vision.py | Damage description generation | LLaVA-1.5-7B Q4_K_M (4GB) |
| Structuring | json_structurer.py | Natural language β JSON | Swallow-8B (Ollama) |
| Scoring | priority_scorer.py | Repair priority calculation | Rule-based (YAML) |
| Pipeline | end_to_end.py | Orchestration | Python 3.12 |
Processing Time Breakdown (per image):
βββββββββββββββββββββββββββββββββββββββββββ
β Preprocessing: ~2s (4%) β
β Vision Analysis: ~42s (81%) β
β JSON Structuring: ~5s (10%) β
β Scoring: <1s (2%) β
βββββββββββββββββββββββββββββββββββββββββββ€
β Total: ~51.6 seconds/image β
βββββββββββββββββββββββββββββββββββββββββββ
3 Vision Modes Implemented
Complete Pipeline
Validation Tests
Windows Encoding Issues Resolved
Comprehensive evaluation of LLaVA-1.5-7B quantization levels (Q4_K_M, Q5_K_M, Q8_0) to determine the optimal balance between accuracy, speed, and model size for bridge damage assessment.
Developed a 5-point quality scoring system (v0.2 design) to assess damage description completeness:
| Component | Max Points | Criteria |
|---|---|---|
| Damage Types | 2.0 | Recognition of crack, rebar exposure, corrosion, spalling, section loss |
| Severity Level | 1.0 | Minor, moderate, severe classification |
| Location Info | 1.0 | Spatial information (top, bottom, left, right, etc.) |
| Extent Info | 1.0 | Coverage description (local, widespread, partial, etc.) |
| Total | 5.0 | Comprehensive damage assessment score |
| Quantization | Model Size | Init Time | Avg Inference | Quality Score | Text Length | Success Rate |
|---|---|---|---|---|---|---|
| Q4_K_M | 4.1GB | 3.6s | 5.43s | 2.93 Β± 1.53 | 168 Β± 41 | 254/254 (100%) |
| Q5_K_M β | 4.8GB | 4.5s | 5.67s | 3.18 Β± 1.35 | 160 Β± 37 | 254/254 (100%) |
| Q8_0 | 7.2GB | 5.9s | 7.63s | 3.27 Β± 1.39 | 162 Β± 39 | 254/254 (100%) |
Performance Comparison:

Figure 1: Performance metrics across quantization levels (N=254). Shows average inference time, model size vs total time, quality score comparison, and summary statistics.
π English Version | Japanese Version

Figure 2: Comprehensive statistical analysis including violin plots (quality, text length, inference time), box plots, scatter plots (quality vs text length/inference time), and detailed statistics table.
Violin Plot Analysis:
Statistical Significance (Mann-Whitney U Test):
| Quantization | Correlation Coefficient | Interpretation |
|---|---|---|
| Q4_K_M | -0.559 | Moderate negative (longer β lower quality) |
| Q5_K_M | -0.148 | Weak negative (stable across lengths) |
| Q8_0 | -0.393 | Moderate negative |
Insight: Q5_K_M maintains consistent quality regardless of description length, indicating robust performance.
Speed-Quality Efficiency:
Q4_K_M: 0.54 quality/sec (2.93 / 5.43)
Q5_K_M: 0.56 quality/sec (3.18 / 5.67) β BEST β
Q8_0: 0.43 quality/sec (3.27 / 7.63)
Best Quality-Speed Balance
Stable Performance
Resource Efficiency
| Priority | Quantization | Use Case |
|---|---|---|
| π₯ Recommended | Q5_K_M | Production deployment (best balance) |
| π₯ Alternative | Q8_0 | High-accuracy applications (if speed is not critical) |
| π₯ Development | Q4_K_M | Fast prototyping only |
For Bridge Damage Assessment:
Quantization is Not Free
Sweet Spot Exists
Statistical Validation is Essential
GPU Compatibility Matters
Quality Metrics Enable Optimization
Domain-specific fine-tuning of LLaVA-1.5-7B (Q5_K_M) using QLoRA (Quantized Low-Rank Adaptation) to improve damage description quality from 3.18/5.0 β 4.0+/5.0 while maintaining inference speed.
Based on v0.2 results:
Q4_K_M rejected due to bimodal quality distribution (high variance). Q8_0 rejected due to poor cost-benefit ratio.
Base Model: LLaVA-1.5-7B (Q5_K_M quantization)
Adapter: LoRA (Low-Rank Adaptation)
Target Modules:
- q_proj, v_proj (attention layers)
- vision_tower (optional, for domain-specific visual features)
LoRA Rank: 16-32 (balance between capacity and efficiency)
LoRA Alpha: 32-64 (scaling factor)
Dropout: 0.05-0.1 (prevent overfitting)
Quantization: 4-bit (during training, base model frozen)
Source: 254 rebar exposure images + human-annotated ground truth
Data Preparation:
High-Quality Subset Selection (N=50-100):
Annotation Format:
{
"image": "kensg-rebarexposureRb_001.png",
"ground_truth": "Severe rebar exposure with extensive corrosion covering approximately 60% of the bottom-right beam section. Multiple horizontal cracks (3-5mm width) extend across the surface. Significant concrete spalling reveals rebar with heavy rust accumulation. Section loss estimated at 15-20mm depth.",
"quality_score": 5.0,
"damage_types": ["rebar_exposure", "corrosion", "crack", "spalling", "section_loss"],
"severity": "severe",
"location": "bottom_right_beam",
"extent": "60%"
}
Data Augmentation:
| Parameter | Value | Rationale |
|---|---|---|
| Batch Size | 4-8 | Fit in 16GB VRAM (gradient accumulation if needed) |
| Learning Rate | 1e-4 to 5e-4 | Typical for LoRA fine-tuning |
| Epochs | 3-5 | Avoid overfitting on small dataset |
| Optimizer | AdamW | Standard for transformers |
| LR Schedule | Cosine with warmup | Smooth convergence |
| Gradient Clipping | 1.0 | Stability |
| Mixed Precision | FP16 | Memory efficiency |
graph LR
A[254 Images] --> B[Manual Annotation<br/>50-100 samples]
B --> C[Train/Val Split<br/>80/20]
C --> D[QLoRA Training<br/>Q5_K_M base]
D --> E[Adapter Weights<br/>~10-50MB]
E --> F[Merge with Base<br/>Optional]
F --> G[Validation<br/>254 images]
G --> H{Quality > 4.0?}
H -->|Yes| I[Deploy v0.3]
H -->|No| J[Hyperparameter Tuning]
J --> D
| Metric | v0.2 (Q5_K_M) | v0.3 (Fine-Tuned) | Improvement |
|---|---|---|---|
| Quality Score | 3.18 Β± 1.35 | 4.0-4.5 Β± 0.8 | +26-41% |
| Damage Type Coverage | 2.93/4 keywords | 3.5-4.0/4 | +19-37% |
| Severity Accuracy | 65% | 85-90% | +31-38% |
| Location Precision | 58% | 80-85% | +38-47% |
| Extent Quantification | 42% | 70-75% | +67-79% |
| Inference Speed | 5.67s | 5.5-6.0s | ~0% (maintained) |
β Minimum Acceptable Performance:
π― Target Performance:
peft, bitsandbytes, transformers)# QLoRA fine-tuning
pip install peft>=0.11.0 # Parameter-Efficient Fine-Tuning
pip install bitsandbytes>=0.43.0 # 4-bit quantization
pip install transformers>=4.41.0 # HuggingFace transformers
pip install accelerate>=0.30.0 # Distributed training utilities
# Training monitoring
pip install wandb # Experiment tracking (optional)
pip install tensorboard # Local logging (optional)
# Data preparation
pip install datasets # HuggingFace datasets
Option A: Fine-tune FP16 model β Convert to GGUF
llama.cpp toolsOption B: Use unsloth (optimized QLoRA)
unsloth library: 2x faster training, 50% less VRAMOption C: Cloud training (if local VRAM insufficient)
| Risk | Mitigation Strategy |
|---|---|
| Overfitting (small dataset N=50-100) | Strong regularization (dropout 0.1), early stopping, data augmentation |
| Catastrophic forgetting | Low learning rate (1e-4), small LoRA rank (16-32), freeze base model |
| Inference slowdown | Profile adapter overhead, consider merging weights, quantize adapter |
| Annotation quality | Inter-rater agreement (Cohen's kappa >0.7), expert review |
| GGUF compatibility | Start with HF model, convert after training, validate outputs |
| Test Scale | Processing Time | Success Rate | Avg Time/Image |
|---|---|---|---|
| Single Image | 42s | 100% | 42s |
| 10-Image Batch | 8m 35s | 100% | 51.6s |
| 50-Image (Est.) | ~43m | - | ~52s |
| 254-Image (Est.) | ~3.6h | - | ~51s |
git clone https://github.com/your-username/damage_text_score.git
cd damage_text_score
# Windows PowerShell
python -m venv .venv
.venv\Scripts\Activate.ps1
# Linux/macOS
python -m venv .venv
source .venv/bin/activate
# PyTorch (CUDA 12.4)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
# llama-cpp-python (GPU version)
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124
# Other dependencies
pip install -r requirements.txt
# LLaVA GGUF Model (Recommended)
python download_llava_gguf.py
# Downloads:
# - models/ggml-model-q4_k.gguf (4.08GB)
# - models/mmproj-model-f16.gguf (624MB)
# Install Ollama
# https://ollama.com/download
# Pull Swallow-8B model
ollama pull swallow8b-lora-n4000-v09-q4:latest
# Test single image (~42s)
python quickstart.py --mode 1
# Process 10-image batch (~8.5 min)
python quickstart.py --mode 2
# Process 50 images (~43 min)
python quickstart.py --mode 3
# Process all 254 images (~3.6 hours)
python quickstart.py --mode 4
data/outputs/
βββ quickstart_single.csv # Single image result
βββ quickstart_10images.csv # 10-image results
βββ quickstart_50images.csv # 50-image results
βββ quickstart_254images.csv # Full dataset results
CSV Example:
image_name,damage_type,severity,location,risk,priority_score,priority_level,description
kensg-rebarexposureRb_001.png,crack,high,girder,structural,0.952,5,Extensive cracking observed...
JSON Structure:
{
"damage_type": "rebar_exposure",
"severity": "high",
"location": "girder",
"risk": "structural",
"description_ja": "ιηι²εΊγθ¦γγγθ
ι£γι²θ‘γγ¦γγ...",
"key_features": ["rebar exposure", "moderate corrosion"],
"priority_score": 0.952,
"priority_level": 5
}
from src.pipeline.end_to_end import DamageAnalysisPipeline
# Initialize pipeline
pipeline = DamageAnalysisPipeline("config.yaml")
# Process single image
result = pipeline.process_image("path/to/image.png")
# Batch processing
results = pipeline.process_batch(image_paths, output_csv="results.csv")
| Mode | Model | Size | Time/Image | GPU Usage | Rating |
|---|---|---|---|---|---|
| llama-cpp-python | LLaVA-1.5-7B Q4_K_M | 4.08GB | 51.6s | 100% | βββββ |
| HuggingFace | llava-1.5-7b-hf | 14GB | 45s | 100% | ββββ |
| Ollama | llava:7b | 4.7GB | 88s | 0% (CPU) | ββ |
llama-cpp-python (Recommended)
HuggingFace
Ollama
LLaVA-1.5-7B - Vision-Language Model
Swallow-8B - Japanese LLM
damage_text_score/
βββ .venv/ # Python virtual environment
βββ data/ # Dataset
β βββ images_human_inspect_n254/ # Input images (254 files)
β βββ preprocessed/ # Preprocessed images
β βββ outputs/ # Processing results
β βββ descriptions/ # Vision outputs
β βββ structured/ # JSON structured outputs
β βββ scores/ # Scoring results
βββ models/ # Model files
β βββ ggml-model-q4_k.gguf # LLaVA GGUF (4.08GB)
β βββ mmproj-model-f16.gguf # MMProj (624MB)
β βββ scoring_rules.yaml # Scoring rules
βββ src/ # Source code
β βββ preprocessing/ # Preprocessing module
β β βββ image_preprocessor.py
β βββ vision/ # Vision analysis
β β βββ llama_cpp_vision.py # llama-cpp-python (Recommended)
β β βββ granite_vision.py # HuggingFace version
β β βββ ollama_vision.py # Ollama version
β βββ structuring/ # JSON structuring
β β βββ json_structurer.py
β βββ scoring/ # Scoring
β β βββ priority_scorer.py
β βββ pipeline/ # Pipeline orchestration
β β βββ end_to_end.py
β βββ utils/ # Utilities
β βββ config.py
β βββ ollama_client.py
βββ config.yaml # System configuration
βββ quickstart.py # Quick start script
βββ download_llava_gguf.py # Model download script
βββ requirements.txt # Python dependencies
βββ README.md # This file (English)
βββ README_JP.md # Japanese documentation
βββ CHANGELOG.md # Version history
βββ LICENSE # MIT License
Symptom: Japanese characters appear garbled in PowerShell
Solution:
# Change to UTF-8
chcp 65001
python quickstart.py
Symptom: CUDA out of memory error
Solution:
# config.yaml
llama_cpp_vision:
n_gpu_layers: 20 # Reduce from -1 (all layers) to partial GPU
Symptom: Failed to connect to Ollama
Solution:
# Check Ollama server
ollama list
# Restart server
ollama serve
Symptom: Failed building wheel for llama-cpp-python
Solution:
# Install CUDA version explicitly
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124
# Or enable CUDA via environment variable
$env:CMAKE_ARGS="-DLLAMA_CUBLAS=on"
pip install llama-cpp-python --force-reinstall --no-cache-dir
If you use this project in your research, please cite:
@software{bridge_damage_assessment_2026,
title = {Bridge Damage Assessment and Repair Priority Scoring System},
author = {Your Name},
year = {2026},
version = {0.1.0},
url = {https://github.com/your-username/damage_text_score}
}
MIT License - See LICENSE for details
Last Updated: March 20, 2026 (v0.1.0)