Hybrid Vision Pipeline

This repository explores a split-compute approach for object detection and segmentation. A lightweight edge container runs YOLOv8-n-seg on CPU and sends compressed features to a heavier cloud container with GPU access. The aim is to measure accuracy, bandwidth and latency trade-offs while experimenting with a reinforcement-learning controller that decides when to offload processing.

Pipeline Overview

Edge Detection (src/edge/detector.py)

• Runs YOLOv8-n-seg using only CPU resources.
• Taps the neck feature map before the segmentation head.
• Compresses that tensor with the codec in src/common/codec.py (baseline zlib or the Adaptive Flow Encoder).
• Packages metadata and compressed bytes into a single MessagePack blob.

Cloud Segmentation (src/cloud/segmenter.py)

• Receives the blob and decompresses the feature tensor.
• Reconstructs masks with the full Gen2Seg pipeline or, optionally, the original YOLOv8n-seg decoder.
• Returns masks, timing metrics and the original header.

Adaptive Split Controller (src/common/offload_policy.py)

• Implements threshold and RL-based policies that decide per frame whether to keep computation local or offload.
• Works with traffic control scripts in tc/ that emulate 5G, 4G and 3G links so the entire system can be evaluated on a single machine.

Research Goals

• Model and codec sizing – determine how small the edge model can be and how aggressively features can be compressed while maintaining target mAP on COCO-val.
• Codec efficiency – measure latency and energy cost of the flow-based codec versus the bandwidth saved.
• Adaptive split – evaluate the RL controller that chooses between local processing and offloading based on bandwidth and queue depth.

The experiments run inside Docker containers via docker-compose.yml. The edge container can be throttled to Raspberry Pi‑class resources (for example an 8 GB Pi 5). If time permits, the same container can run on a real Pi to validate the concept on physical hardware.

Development

1. Install the Python dependencies from requirements-dev.txt.

2. Set up Python path: The source code is organized under src/. To run modules directly (without the src. prefix), add the src directory to your Python path:

   Windows (PowerShell):

   $env:PYTHONPATH = "c:\path\to\hybrid-vision\src"
   

   Linux/macOS:

   export PYTHONPATH="/path/to/hybrid-vision/src"
   

3. (Optional) Clone the Gen2Seg repo if you want to use the Gen2Seg decoder outside Docker:

   python hv.py fetch-gen2seg
   export PYTHONPATH="$(pwd)/gen2seg:${PYTHONPATH}"
   

4. Run pytest to execute the unit tests (see examples below for running specific tests).

5. Both the edge and cloud containers have their own requirements-*.txt files and Dockerfiles under docker/.

Running Tests

pytest                             # run the full suite
pytest tests/test_codec.py         # run a single test file
pytest tests/test_codec.py::test_encode_zlib  # run one test case

Training the Adaptive Flow Encoder (AFE)

The AFE is an invertible flow that can compress either mid-level features or full images more efficiently than zlib. Separate checkpoints are produced for each kind of data.

Step 1: Collect a training dataset (on dev machine)

For feature compression, dump neck tensors from COCO validation images:

python -m common.dump_neck_tensors --imgs datasets/coco/val2017 --out results/neck_stack.pt --count 2000

This extracts neck features from 2000 COCO images and saves them as a PyTorch tensor file.

Step 2: Train the AFE model (on dev machine)

Train on features (default) or on raw images by specifying --kind image:

# Feature codec
python -m src.training.train_afe --tensors results/neck_stack.pt --epochs 20

# Image codec
python -m src.training.train_afe --tensors results/image_stack.pt --kind image

Feature weights are written to _weights/codec.pt and image weights to _weights/image_codec.pt.

Step 3: Deploy for experiments (in Docker)

1. Mount the trained model(s) into your Docker containers:

   • feature codec → /app/weights/codec.pt
   • image codec → /app/weights/image_codec.pt

2. Set the codec backend to use AFE instead of zlib:

   export HYBRID_CODEC=afe
   

3. Run your experiments with the trained AFE codec:

   docker-compose up
   # Then run experiments as normal - they will automatically use the AFE codec
   

Testing locally (alternative)

You can also test the AFE codec locally on your dev machine by:

1. Copy the trained model to the expected location:

   mkdir -p /app/weights  # or adjust _ckpt path in codec.py
   cp _weights/codec.pt /app/weights/codec.pt
   

2. Set the environment variable and run experiments:

   $env:HYBRID_CODEC = "afe"  # PowerShell
   python -m experiments.experiment_runner --dataset datasets/coco --frames 200 --csv results/afe_test.csv
   

Note: The codec automatically switches between zlib (default) and AFE based on the HYBRID_CODEC environment variable.

Model Pruning & Quantization

You can prune and quantize the YOLO weights for edge deployment with the helper script under src/training. The tool supports dynamic, static and quantization-aware training (QAT) modes:

# direct module invocation
python -m training.optimize_yolo yolov8n-seg.pt pruned_quantized.pt --prune 0.2 --quant static

# hv.py shorthand
python hv.py optimize yolov8n-seg.pt pruned.pt --prune 0.3 --quant static

By default dynamic quantization is used. Pass --quant static for post-training static quantization or --quant qat (optionally with --steps) to run a short QAT loop before converting the model.

To prune the provided YOLO weights and export a quantized ONNX model:

python -m training.prune_quantize_export _weights/yolov8n-seg.pt

Running Experiments

The experiment runner processes a batch of COCO images through the edge → cloud pipeline and records latency, bandwidth and accuracy metrics. Use --policy heuristic (or --policy rl) together with --bandwidth to enable bandwidth-aware routing.

The RL policy can be trained online and persisted for later runs. Pass --policy-train together with a --policy-path to update and save the Q-table while processing frames:

python -m experiments.experiment_runner --dataset datasets/coco --frames 200 \
       --policy rl --bandwidth 8 --policy-train --policy-path policy.pkl

To reuse a previously trained policy, supply the same path without the --policy-train flag:

python -m experiments.experiment_runner --dataset datasets/coco --frames 200 \
       --policy rl --bandwidth 8 --policy-path policy.pkl

# direct module invocation
python -m experiments.experiment_runner --dataset datasets/coco --frames 200 \
       --profile 5G --policy heuristic --bandwidth 8 --csv results/run1.csv

# hv.py shorthand
python hv.py experiment --dataset datasets/coco --frames 200 --profile 5G \
       --policy heuristic --bandwidth 8 --csv results/run1.csv

Factorial Experiment

To reproduce the full study, sweep the four quantisation modes, three compression ratios, four bandwidth classes and two control policies using the convenience script below. The aggregated summary for all 96 configurations is written to results/factorial.csv.

python -m experiments.factorial_experiment --dataset datasets/coco --frames 1000 --out results/factorial.csv

The accompanying analysis notebook demonstrates how to compute ANOVA tables, mixed‑effects models and the headline numbers used in the dissertation text:

jupyter lab notebooks/factorial_analysis.ipynb

hv.py Shorthand

The hv.py script in the repository root exposes handy commands so you don't have to remember full module paths:

# Run the edge detector on one image
python hv.py edge path/to/image.jpg

# Run the complete edge → cloud pipeline
python hv.py cloud path/to/image.jpg

# Launch the docker containers
python hv.py compose-up

Run python hv.py -h to see all available subcommands.