Vuga AAC

An offline Kinyarwanda picture-to-speech communication app for disabled children who cannot speak or are hard to understand, with machine-learning next-word suggestions.

Description

Who it is for. Many disabled children cannot use speech to ask for help, make a choice, say "no" or join a conversation. Others speak, but people outside the family cannot understand them. This includes children with:

• autism
• cerebral palsy
• Down syndrome
• intellectual disability
• speech motor disorders such as childhood apraxia of speech

Rwanda's child census identifies children with communication difficulties (NISR children report).

Problem. Augmentative and alternative communication (AAC) lets these children communicate with pictures and a device that speaks for them (ASHA AAC overview). In Rwanda, though, most families and schools cannot use AAC apps:

• they are mostly in English, not Kinyarwanda
• they are expensive
• they often need dedicated devices or a constant internet connection

Without a way to communicate, a child's needs and choices are easily missed. Frustration can show up as behaviour, and the child is left out of class and family life.

Solution. Vuga ("speak" in Kinyarwanda) is a free communication app in Kinyarwanda. It runs on an ordinary Android tablet or phone and works fully offline after the first load.

• The child taps picture cards to build a message. Vuga speaks it aloud in Kinyarwanda, and Erekana ("show") displays it full screen.
• There are 181 everyday words, such as "want", "help", "no", "water", "toilet", "hurt" and "play". Each word has a picture and a Kinyarwanda voice clip.
• A parent, teacher or therapist sets up the app for each child in the caregiver area, Umurezi, which is protected by a PIN.
• All data stays on the device unless a caregiver chooses to sync it.

How it adapts to each child's disability. Each child has their own profile, and the caregiver adjusts the board to fit that child:

Not yet supported. These are planned or open problems:

• Switch scanning for children who cannot tap a screen, for example some children with severe cerebral palsy.
• Protection against accidental taps for children with tremor.
• Reading a tile's word aloud before it is selected, for blind or low-vision children.
• Testing with disabled children, families and therapists. This has not happened yet. Partner organisations working with disabled children in Rwanda are the next step.

Machine learning. A next-symbol predictor suggests likely next words in a separate row, so the main board's layout never moves. The model notebook (HTML version) covers data engineering, visualisations, model architecture and metrics. It compares three models trained on Kinyarwanda sentences mapped to the board vocabulary:

The app ships the small trigram model because it is almost as accurate as the GRU, runs instantly offline, and is only about 100 KB.

Architecture.

• App: React + Vite PWA with Dexie (IndexedDB) storage and a service-worker cache.
• API (optional): FastAPI with Swagger UI, SQLAlchemy over SQLite or PostgreSQL, JWT authentication, sync, summaries and /predict.
• ML package and notebook: in ml/.

Status (12 October 2026). This is an initial demonstration.

• The interface, pictures, speech, parent access and prediction work, and are covered by automated and browser tests.
• The speech is synthetic (Meta MMS-TTS) and has not yet been reviewed by a native speaker.
• No school, caregiver or therapist has evaluated the app yet, and no participant findings are claimed.

The app's labels, dialogs, accessible names and messages are in Kinyarwanda. Developer documentation is in English.

Set up the environment and run the project

Prerequisites:

• Git
• Node.js 20 with npm, for the app
• Python 3.11+, for the notebook and API (optional)

1. Get the code

git clone https://github.com/jbyiringiro/capstone-project.git
cd capstone-project

If you downloaded the ZIP from GitHub instead, unzip it and open a terminal in the extracted folder.

2. Run the app (communication board)

cd app
npm ci              # install dependencies
npm test            # run the unit tests
npm run build       # production build with offline service worker
npm run preview -- --host 127.0.0.1

Open http://localhost:4173. For development with hot reload, run npm run dev and open http://localhost:5173. On Windows PowerShell, use npm.cmd if the execution policy blocks npm.

Using the app:

• The board opens directly, with no account or PIN. Tap pictures to build a message, then press Vuga to hear it or Erekana to show it full screen.
• Hold Umurezi (or activate it by keyboard) for the caregiver area. On first use, a parent chooses a four-to-six-digit PIN. Setup can be dismissed without blocking the child.
• Parent tabs are Abana (profiles), Amashusho (pictures), Ikibaho (board), Ijwi (speech), Incamake (summary), Ohereza (export/sync) and Kwinjira (change PIN).

3. Set up Python, the notebook and the API (optional)

From the repository root:

python -m venv .venv
# Windows: .venv\Scripts\activate    macOS/Linux: source .venv/bin/activate
python -m pip install -e "./ml[dev,notebook]" -e "./api[dev]"

python ml/notebooks/run_notebook.py          # re-run the model notebook
python -m pytest ml/tests api/tests          # run the Python tests
python -m uvicorn app.main:app --app-dir api --host 127.0.0.1 --port 8000

Open Swagger UI at http://localhost:8000/docs.

API configuration. Copy .env.example to .env to configure the API:

• Set VUGA_MODEL_DIR to ml/artifacts.
• Set VUGA_CORS_ORIGINS to the app's origin.
• For persistent sessions, set VUGA_JWT_SECRET to a private value of at least 32 characters. Without it, local development uses a temporary key.

To run the API with PostgreSQL in Docker instead, set VUGA_JWT_SECRET and run docker compose up.

The notebook reads the committed ml/notebooks/evidence/ snapshot. It does not download the corpus or retrain the neural models. To regenerate the trigram model from that snapshot, run python ml/scripts/export_artifacts.py --data ml/notebooks/evidence. Optional neural tests need ml[neural].

More detail: project guide.

Designs

Screenshots of the working app (from docs/screenshots/licensed-2026-10-10):

Figma mockups and design system (in design/):

• 16 screen mockups (tablet and phone) as Figma-importable SVGs in design/figma-import/, with PNG previews in design/previews/. For example: all screens, tablet board, compose, mobile board.
• Style foundations (foundations.html, foundations sheet), design tokens (tokens.json) and 25 custom icons (icon sheet).
• An interactive clickable preview: open design/preview.html in a browser.

The mockups record the earlier visual design. The running app (screenshots above) is the current version, with Kinyarwanda text and licensed photographs.

Circuit diagram. Not applicable. Vuga is a software project that runs on existing tablets and phones. The physical-access parts in hardware/ (a 3D-printable keyguard) and device/ (firmware scaffold for a possible talker box) are early drafts and have not been built.

Deployment plan

The full plan is in docs/deployment_plan.md. In summary:

1. Phase 0, local preview (done): npm run build and npm run preview, as shown in the video demo.
2. Phase 1, public app: publish the static PWA (app/dist) on Netlify or Cloudflare Pages over HTTPS. The board needs no server, so this alone makes it usable. A tablet installs the app once and then works offline.
3. Phase 2, optional API: deploy api/Dockerfile to Render with managed PostgreSQL, secrets set as environment variables, CORS limited to the app's domain, and a /health check.
4. Phase 3, pilot: only after a native speaker reviews the speech and caregivers review the vocabulary and pictures.

The plan also covers configuration, checks after each deploy, rollback, privacy for children's data, licensing and cost ($0 for the demo).

Code files

Key source files:

Pictures and Kinyarwanda speech

Pictures. Each card uses a licensed online image, bundled locally in app/public/photos/ so it works offline. The photo manifest records source pages, authors, licences and hashes. These are not photographs of project participants. Umurezi → Ikibaho → Amafoto afatika switches a profile to ARASAAC pictograms instead. Fetch those with python ml/scripts/fetch_pictograms.py; they have their own licence.

Speech. The app bundles 181 synthetic Kinyarwanda WAV clips, generated from the exact label_rw entries in vocab/symbols.csv with Meta AI's pinned facebook/mms-tts-kin model. They have not been reviewed by a native speaker.

• Playback order: a caregiver's saved recording, then an installation's /audio/SYMBOL.wav, then the bundled /audio/rw/SYMBOL.wav, then a Kinyarwanda system voice if one is installed. English is never used as a fallback.
• If speech fails, the message is kept and Erekana is offered.
• Caregiver recordings: in Umurezi → Ijwi, a caregiver can record or upload, preview, save or delete a recording for any word. Recordings stay on the device and are excluded from export, sync and the repository.

Details: audio setup and speech licensing and generation.

Parent access security.

• Parent access locks when the user returns to the board, reloads the page, leaves the app in the background, or makes no input for five minutes.
• Five failed PIN attempts start a 30-second cooldown.
• Only a salted PBKDF2-derived credential is stored.

This PIN is a local guard against accidental settings changes. It is not a server account.

Verification

The current implementation record has the executed checks and screenshots. They cover:

• asset provenance and hashes
• offline use of images and audio
• child communication
• PIN setup, unlock and change
• cooldown persistence
• draft preservation
• automatic locking

The API and ML results are from September; the later interface changes did not retrain the predictor.

This repository includes the bundled pictures, Kinyarwanda speech, the trigram model and the executed notebook. It does not include dependencies (node_modules, .venv), model weights or private data; install those with the setup steps above.

Licences

No held-out test examples or participant data are included.