---

Overview

White canes sense about a metre ahead at ground level, and GPS apps stop working at the door. Indoors (universities, hospitals, offices) a visually impaired person still has to guess where the chair, the open door or the person in the corridor is.

Raahi turns any Android phone into a guide. The phone streams its camera to a Raahi server. The server recognises objects, measures how far away they are, decides the safest way forward and answers with short spoken instructions in a natural female voice, in Urdu or English:

"کرسی سامنے ہے، دو قدم پر۔ دو قدم بائیں ہٹیں۔" "Chair ahead, two steps away. Move two steps left."

The whole app can be used without looking at the screen. Say "Raahi" followed by a command to open any screen, change settings or control guidance, in Urdu, Roman Urdu or English.

Raahi complements the white cane; it does not replace it.

Key features

Architecture

Each camera frame goes through object detection and depth, then the guidance engine and announcer. The reply is one Protocol 5.2 result: Urdu clip IDs, overlay boxes, haptics, and left / centre / right zones. The judge dashboard shows the live camera, depth map, and those distances.

User flow

Walk with Raahi. Say "Raahi, start", hold the phone upright, and the camera names what is ahead with its distance. A spoken instruction in Urdu or English tells the user how to step aside. If something is too close, Raahi says "Stop!" and then the way out. When the path is clear, guidance confirms it. A question asked mid-walk pauses guidance, then it resumes.

Control everything by voice. "Raahi, settings kholo" is recognised, understood on the phone, and the app acts (theme, language, navigation) and confirms with "Okay". Works in Urdu and English, and complements the white cane.

How it works

 Android phone                                  Raahi server (FastAPI)
 ─────────────                                  ─────────────────────
 CameraX frame (upright, 416 px JPEG) ───WS───▶ YOLOE-26 object detection ─┐
                                                Depth Anything V2 (metres) ─┤
                                                                            ▼
                                                Guidance engine
                                                  · left / centre / right zones
                                                  · nearest obstacle, distance in steps
                                                  · safest side + steps to move
                                                  · find / describe modes
                                                  · smoother + announcer (no repeats,
                                                    stop once, then the way out)
 Plays prerecorded clips (Urdu / English) ◀──WS── clip ids: ["obj_chair","pos_ahead",
 vibrates on danger                               "steps_2","move_left_2"]
 Mic ─▶ ElevenLabs Scribe ─▶ on-device command matcher ─▶ navigation / settings / guidance

Why prerecorded clips? Guidance has to be in sync with what the camera sees. Live text-to-speech adds hundreds of milliseconds and needs the network. Raahi assembles sentences from about 100 short recorded phrases per language (object, position, distance, action), so audio starts as soon as a decision is made. A cached live-TTS fallback covers rare labels.

Why fuse depth and object size? Monocular depth is excellent at room scale but unreliable very close to the camera. Raahi combines the depth map with each detected object's known real-world size (height, or width when the object is cut off by the frame) and keeps the closer estimate, so a person standing right in front of the camera is never reported as "path clear".

Tech stack

Repository structure

Raahi/
├── backend/
│   ├── server/
│   │   ├── main.py              FastAPI app: /ws (phone), /dashboard, /health, /calibrate
│   │   ├── perception/          detector.py (YOLOE), depth.py (Depth Anything V2), sizes.py
│   │   ├── engine/              navigator, find mode, describe, smoother, announcer, geometry
│   │   ├── voice/               intents, Scribe token, live TTS, app command reference
│   │   ├── dashboard/           judge dashboard (HTML + live WebSocket)
│   │   └── mocks/               mock detector / server for offline development
│   ├── voice_assets/            Urdu + English clip library, batch splitter, sentence composer
│   ├── shared/classes.json      detector classes, sizes and clip mapping (single source)
│   ├── models/                  exported detector models
│   ├── training/                export + fine-tuning notebooks
│   ├── tools/                   live laptop demo, benchmarks, calibration, model export
│   └── tests/                   pytest suite (engine, voice, commands, server)
├── frontend/
│   ├── android/                 native Android app (Kotlin + Jetpack Compose)
│   └── web-prototype/           early web prototype and on-device detector plugin
└── docs/                        project plan, protocol contracts, design notes

Getting started

Backend

Requires Python 3.11+.

cd backend
python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install torch torchvision                          # CPU-only machines: add --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt
cp .env.example .env                                   # add ELEVENLABS_API_KEY and ELEVENLABS_VOICE_ID
python tools/download_models.py                        # depth weights (once)
python -m server                                       # phone: ws://<server-ip>:8000/ws

• Dashboard: http://<server-ip>:8000/dashboard
• Tests: python -m pytest
• Laptop demo without a phone (webcam, depth, guidance, voice): python -m tools.live_demo --lang en (or --lang ur)

Android app

1. Open frontend/android/ in Android Studio (JDK 17).
2. Build and run on a phone on the same network as the server.
3. In the app go to Settings › Connection and enter http://<server-ip>:8000, then Test connection.
4. Grant camera and microphone access during onboarding.

Voice commands

Start with "Raahi" (hands-free) or long-press anywhere (no wake word needed).

The vocabulary is a single file (frontend/android/app/src/main/assets/voice_commands.json). New phrases can be added without code changes.

Team

Roadmap

• Fully on-device mode (detection already exports to LiteRT/TFLite)
• Stairs and drop-off detection
• Outdoor navigation
• More languages (Punjabi, Pashto, Sindhi)
• Partnerships with schools and organisations for the visually impaired

---