👓 BSmart — Multimodal AI Pipeline for Smart Glasses

A lightweight, high-performance multimodal AI system designed for smart glasses to assist visually impaired individuals with voice interaction, visual question answering, and autonomous navigation assistance.

---

✨ System Architecture & Features

---

📁 Repository Structure

BSmart/
├── backend/                     # Python FastAPI Backend & AI Pipelines
│   ├── server.py                # FastAPI REST server for workstation/reference testing
│   ├── pipelines/               # Core multimodal processing pipelines
│   │   ├── voice_control.py     # Voice command recognition & activation
│   │   ├── visual_qa.py         # Visual QA multimodal chatbot pipeline
│   │   └── autopilot.py         # Reference obstacle detection & depth guidance
│   ├── scripts/                 # Utility, setup & diagnostic scripts
│   │   ├── check_glass.py       # 11/11 Sanity & inference verifier
│   │   ├── download_models.py   # Pretrained AI models downloader
│   │   ├── setup_glass.sh
│   │   ├── start_backend.bat
│   │   └── start_backend.sh
│   └── requirements.txt         # Python dependencies
│
├── mobile_app/                  # React Native mobile application
│   ├── android/                 # Android project with ONNX Runtime and native Kotlin modules
│   ├── src/                     # UI components, state machine & inference services
│   └── scripts/                 # Icon generation & dependency patches
│
├── firmware/                    # Smart Glasses ESP32-S3 hardware code
├── ai_core/                     # ONNX model export & packaging scripts
├── assets/                      # App icons (icon.svg) & sound clips (Open_f1..f4.mp3)
├── docs/                        # Architecture & documentation
├── test/                        # Test photos & audio clips
└── start_backend.bat            # One-click Windows backend server startup

---

🚀 Quick Setup & Installation Guide

1. Environment Activation

# Clone the repository
git clone https://github.com/GiaPhaZZZ/BSmart.git
cd BSmart

# Activate Virtual Environment (WSL / Ubuntu / Windows)
source ./glass/bin/activate  # or ./glass/glass/bin/activate

2. Download Pretrained Models

Download all required baseline weights (yolo26s.pt, PhoWhisper, EnViT5, SmolVLM2, Piper vi_VN):

python backend/scripts/download_models.py

3. INT8 Model Optimization (Must-Do for CPU Speed)

Convert speech & translation models to CTranslate2 INT8 format for real-time CPU performance:

# Convert PhoWhisper ASR
ct2-transformers-converter --model vinai/PhoWhisper-tiny --output_dir models/phowhisper-ct2-int8 --quantization int8

# Convert EnViT5 Translation
ct2-transformers-converter --model VietAI/envit5-translation --output_dir models/envit5-ct2-int8 --quantization int8

4. Verify System Readiness (11/11 Sanity Check)

Run the diagnostic script to verify that all libraries and models load correctly:

python backend/scripts/check_glass.py

Expected Output: 11/11 checks passed with no errors.

---

🌐 FastAPI Backend Server API

Start the backend server to serve the Mobile App and Smart Glasses:

# On Windows
start_backend.bat

# On Linux / WSL / Terminal
python backend/server.py

Server runs at: http://0.0.0.0:8000

API Specification

1. GET /health

• Description: Server & model availability status check.
• Response: 200 OK

  {
    "status": "ok",
    "service": "BSmart AI Backend",
    "models": {
      "phowhisper_int8": true,
      "envit5_int8": true,
      "yolo26s": true
    }
  }
  

2. POST /navigate

• Description: Autopilot camera frame obstacle & terrain analysis.
• Request: multipart/form-data (image: JPEG/PNG file)
• Response: 200 OK

  {
    "warning": "Lưu ý, phát hiện car ở phía trước.",
    "detected_objects": ["car"]
  }
  

3. POST /transcribe

• Description: Spoken command recognition.
• Request: multipart/form-data (audio: WAV/MP3 file)
• Response: 200 OK

  {
    "text": "mở tính năng 1.",
    "matched_feature": 1,
    "matched_keywords": ["tính năng 1"]
  }
  

4. POST /qa

• Description: Visual QA Chatbot (Image + Spoken/Text Question -> Answer).
• Request: multipart/form-data (image: file, audio: optional file, question: optional text)
• Response: 200 OK

  {
    "question": "Phía trước có gì?",
    "answer": "Một con phố với hàng cây và những toà nhà dưới bầu trời trong xanh."
  }
  

---

📱 Mobile App & Offline Standalone APK Build

1. Package ONNX Models for Mobile

Export lightweight models to Android assets:

python ai_core/package_models.py

2. Build Debug APK (Standalone Offline)

cd mobile_app/android
./gradlew assembleDebug
adb install -r app/build/outputs/apk/debug/BSmart.apk

3. Build Release APK (Standalone)

cd mobile_app
npx react-native build-android --mode=release

Output APK location: mobile_app/android/app/build/outputs/apk/release/app-release.apk

Android Runtime Notes

• ONNX models are packaged in Android assets, then streamed with a 64 KB buffer into the app sandbox at runtime: filesDir/onnx_models/<model filename>.
• Model files are cached and only recopied when the extracted file size differs from the asset size.
• ONNX Runtime sessions are created from file paths, not in-memory ByteArrays, to avoid heap spikes when loading large models such as smolvlm2_decoder.onnx.
• Sessions are lazy-loaded per active feature and inactive groups can be released through releaseObjectDetectionModels(), releaseVlmModels(), and releaseAsrModels().
• android:largeHeap="true" is not used as the primary memory fix.

Voice Command Aliases

Voice command matching is accent-insensitive and supports common ASR variants:

---

🔌 ESP32-S3 Firmware Flashing

BSmart includes a one-click flashing script that automatically detects the ESP32-S3 COM port, builds from current source when release artifacts are missing or stale, and flashes validated binaries (bootloader.bin, partitions.bin, and firmware.bin).

# Make sure the ESP32 is connected via USB, then run at project folder:
python flash_firmware.py

# Select the port explicitly when more than one ESP32 adapter is present
python flash_firmware.py --port COM7

# Always rebuild from source before flashing
python flash_firmware.py --rebuild

The generated release/firmware/manifest.json records source metadata and SHA-256 hashes for all three binaries. A failed build never falls back to older artifacts. Note: The script automatically installs required dependencies (esptool, pyserial) if they are missing.

---

🧪 CLI Feature Execution Examples

# Function 0: Voice Command Test
python backend/pipelines/voice_control.py --audio test/audio/mo_tinh_nang.mp3

# Function 1: Visual QA Chatbot Test
python backend/pipelines/visual_qa.py --image test/photo/road.jpg --audio test/audio/mieu_ta_khung_canh.mp3 --out out.wav

# Function 3 reference: Autopilot Navigation Test
python backend/pipelines/autopilot.py --image test/photo/road.jpg --audio guide.wav