BAS on-board human activity recognition

Our entry for the ISRO BAS problem statement AI Human Activity Recognition for On-board BAS Experiments. Two parts, each with its own site:

Headline results

HAR assistant, 100 held-out rendered runs it never saw (34 with an injected mistake):

• 34 of 34 mistakes raised a voice alert, each named correctly; 3 of 66 correct runs raised a false alarm; 97% correct verdicts
• alert a median 2.8 s after the wrong step starts
• crew joints tracked to 29 mm in 3D relative to the rack, at any body orientation
• downlink of 687 B/s (3D state + step events) instead of 108 KB/s of video, 158× less
• 5.8 frames/s on a laptop CPU alone (the cameras give 5), 19 frames/s with its RTX 4060

Research, held-out NASA ISS footage:

• full frames 162 KB/s → body-model parameters + 320 px scene image 16.8 KB/s (E1) → scene sent only when it changes 2.2 KB/s (E7), with the same crew meshes on the ground
• the reference body model moves ~120 mm when the crew is rolled 90°; four-rotation canonicalisation brings it to 13 mm (E10)

All results on the HAR side are on synthetic runs from our Blender generator (a mannequin crew member); real-person trials are the next step.

Layout

Not in the repository (large, all regenerable): NASA footage (public domain, images.nasa.gov), Blender renders, datasets, experiment outputs and built sites.

Running

Environments. The HAR assistant runs on Windows or Linux with Python 3.11, torch, ultralytics and opencv-python; voice uses the Windows speech synthesiser, recording and streaming use ffmpeg. The research experiments and the E1/E7/E8 demo need MoGe-2 and SAM 3D Body (we run them in WSL with CUDA; SAM 3D Body cloned at ~/sam3d/sam-3d-body). The generator needs Blender 4.2.

HAR assistant on two camera feeds (indices, video files, stream URLs or folders of frames) and a rack calibration:

python har/run.py --cam0 0 --cam1 1 --calib rack_calibration.json --out sessions/today --voice --gui 8080 --stream udp://192.168.1.20:5000

Add --device cpu to run without a GPU. The console opens at http://127.0.0.1:8080.

Rebuild the HAR models and evaluation

blender -b -P generate.py -- --seeds $(cat render_seeds_train.txt) --fps 2 --no-depth --out render_train
blender -b -P generate.py -- --seeds $(cat render_seeds_test.txt) --fps 5 --no-depth --out render_test
python har/yolo_data.py --renders render_train --out har/data
python har/train_perception.py --data har/data/data.yaml --epochs 24
python har/train_steps.py --data dataset_e11          # needs the 400 generator label runs (generate.py --dry)
python har/evaluate.py --renders render_test --out har/eval

E1/E7/E8 demo: python demo_process.py video.mp4 demo_live/name then serve the folder, or python demo_live_server.py and open http://localhost:8770/ for the live webcam version.

Sites: python viewer.py runs site --standalone and python har/site.py, deployed as static Cloudflare Workers with wrangler deploy (wrangler.jsonc, wrangler.har.jsonc).