SENTINEL — Autonomous Search & Rescue Drone

An autonomous indoor drone that sees, reasons, and acts using NVIDIA Cosmos Reason 2 8B.

The Problem

On February 6, 2023, a 7.8-magnitude earthquake struck Turkey and Syria, killing over 59,000 people and collapsing tens of thousands of buildings. Nearly 240,000 rescuers worked through shifting rubble for two weeks — but most victims who could have been saved were already dead before teams reached them. In the 1985 Mexico City earthquake, 100-135 rescuers were themselves killed by secondary collapses while searching for survivors. The pattern repeats at every scale: in June 2021, Champlain Towers South in Surfside, Florida collapsed, killing 98 people while rescue teams spent weeks navigating underground fires and unstable debris.

The core problem is the same every time: rescuers enter unstable structures blind, with no map, no knowledge of where survivors are, and no warning of what hazards lie ahead.

In the United States alone, 89 firefighters died in the line of duty in 2023, with structural collapse consistently among the leading causes. As recently as February 2025, a firefighter was killed by the collapse of two buildings in New York during search operations. When a building collapses, the survival rate for trapped victims is 90% within the first 24 hours but drops to 20-30% by 72 hours. Every minute of the initial reconnaissance phase — figuring out where survivors are and what hazards exist — is time that both rescuers and victims don't have.

Today, this reconnaissance is done by humans entering unstable structures, often with nothing more than flashlights and listening devices. FEMA's 28 Urban Search & Rescue task forces deploy 70-person teams that take 4-6 hours to mobilize. Research shows that swarms of just five autonomous drones can achieve 90% sensor coverage of a 2 km² area in under 90 minutes — covering ground that would take human teams hours to clear, without putting anyone at risk.

Our Solution

SENTINEL is a fully autonomous quadcopter that flies into collapsed buildings, reasons about what it sees using NVIDIA Cosmos Reason 2 8B, and reports back survivor locations before anyone enters. No human intervention required. The drone explores unknown space, builds real-time 3D maps using SLAM, and makes decisions — where to fly, when to turn, when it has found a survivor — using a vision-language model that sees a camera frame and reasons about it in real-time.

Built for the NVIDIA Cosmos Cookoff by Team Sentinel (Loopworks).

---

Demo Video

https://github.com/user-attachments/assets/89c5fbac-791c-4fc4-9f5e-04ed5633d5de

3/3 survivors found, 0 false positives, fully autonomous. Full quality video (MP4)

---

What SENTINEL Does

Every decision follows the same loop — no human in the loop, no pre-programmed waypoints:

1. SCANNING — Drone hovers, captures a forward-facing camera frame
2. THINKING — Frame + mission context sent to Cosmos Reason 2 8B. The model responds with a <think> reasoning chain and an ACTION: directive
3. EXECUTING — Agent parses the action and flies:
   • MOVE_FORWARD — Fly 3m ahead via direct velocity command (reverse if <1m obstacle)
   • TURN_LEFT / TURN_RIGHT — Rotate 90°
   • MARK_SURVIVOR — Depth-project center pixel to 3D, place red marker at survivor location
   • MARK_HAZARD — Same projection, orange marker for dangerous areas
   • MISSION_COMPLETE — All areas explored, mission ends
4. Repeat — Back to SCANNING until mission completion guards are met (≥60s elapsed, ≥6 scans, ≥3 compass directions explored, ≥2 forward moves)

The drone can't just freeze mid-air waiting for an API response. So the lidar safety bubble runs at 20Hz regardless of API state — if an obstacle is within 0.5m, velocity commands are overridden with emergency repulsion. The drone holds position safely while the model thinks, then executes.

---

Architecture

SENTINEL uses a hybrid control architecture — a Fast Brain for safety and a Slow Brain for reasoning:

Why Hybrid?

Moving from the local 2B quantized model to the full 8B improved reasoning quality dramatically — but introduced 1-3 seconds of network latency per decision. A drone can't wait mid-air for an API response, and it can't fly blind. The hybrid split was the natural solution:

• Fast Brain (Local, 20Hz): Lidar safety bubble prevents collisions regardless of what the API is doing. Three tiers: CRITICAL (stop + repel), WARNING (slow), SAFE (full speed).
• Slow Brain (Cloud, 1-3s): Cosmos Reason 2 8B handles high-level reasoning — where to go, what it sees, whether something is a survivor.
• "Hover and Think": Zero-velocity hover during inference. The drone holds position safely until the response arrives, then executes.

API Resilience

The cloud reasoning path is designed for real-world network conditions:

• Categorized retry — Different exponential backoff for connection errors (8/16/32s) vs timeouts (4/8/16s) vs rate limits (10/20/40s). Client errors (4xx) are not retried.
• Server health gating — 3 consecutive connection failures marks the server unhealthy; health check runs before each subsequent query until it recovers.
• Graceful degradation — If the API dies mid-mission, the drone falls back to safe exploration patterns (turn right) rather than crashing. After sustained failures (>180s, 8+ failures), it completes the mission gracefully.
• Anti-stuck logic — Stagnation detection (<1m movement in 5 cycles), turn-loop prevention (3 consecutive turns → forced MOVE_FORWARD), and emergency fallback keep the drone making progress.

---

Cosmos Reason 2 Integration

SENTINEL uses NVIDIA Cosmos Reason 2 8B as the core decision-making model. This is not post-hoc analysis or monitoring — the model's output directly controls the drone in real-time.

Deployment: Self-hosted via vLLM on a cloud GPU (Nebius), exposed as an OpenAI-compatible API endpoint. The agent connects using the standard OpenAI Python SDK.

Each reasoning cycle:

1. Forward-facing camera frame captured and base64-encoded
2. System prompt built with: mission text, current heading, recent action history, explored/unexplored directions, coverage percentage
3. Sent to Cosmos Reason 2 8B with the frame as a vision input
4. Model responds with <think>[reasoning]</think> followed by ACTION: [CHOICE]
5. Agent parses both — reasoning is displayed on the live dashboard, action is executed

What the model sees: A single RGB frame described as a "thermal imaging feed" (mannequin survivors use green-emissive material to simulate thermal signatures). The prompt tells it: "You are SENTINEL, a Search & Rescue drone AI. Analyze this thermal imaging feed and decide your next action."

What the model decides: One of 6 actions per cycle. The model's <think> chain shows spatial reasoning ("debris on the left, clear path ahead"), threat assessment ("green thermal signature consistent with a person"), and exploration strategy ("I haven't checked the right corridor yet").

---

Key Features

• Closed-Loop Autonomy — Cosmos Reason 2 8B output directly drives physical drone actions. Not post-hoc analysis, not monitoring — real-time control.
• Visible Reasoning — Full <think> chain displayed live on a Rich terminal dashboard. Judges (and first responders) can see why the drone made each decision.
• Real-time 3D SLAM — RTAB-Map fuses RGB-D camera + 3D Velodyne lidar for simultaneous mapping and localization.
• Safety-First Design — Independent 20Hz lidar safety bubble with 3-tier response (CRITICAL/WARNING/SAFE). The drone cannot hit walls even if the network drops.
• Survivor & Hazard Marking — Depth-projected 3D markers placed at detected targets, visible in RViz (red = survivor, orange = hazard). Confidence scoring: ≥0.75 confirmed, ≥0.50 possible.
• Mission Completion Guards — The drone can't declare "mission complete" prematurely — minimum time, scan count, coverage, and directional diversity thresholds must be met.
• Graceful Degradation — API failures trigger safe fallback behaviors, not crashes. The system degrades gracefully from full reasoning to basic exploration to mission completion.

---

The Journey: From Mapping Drone to S&R Agent

This project started as a general-purpose autonomous indoor mapping drone and evolved into SENTINEL through several pivots driven by real engineering challenges.

Foundation (Phases 1-3)

We built from scratch: Isaac Sim 5.1 with a simulated quadcopter, PX4 SITL over Micro-XRCE-DDS, and ROS 2 Humble. The first milestone was getting the TF tree right (map → world → odom → body → base_link) and discovering that Isaac Sim's PX4 implementation has X and Y velocity axes swapped — a bug that took hours of teleop testing to identify.

RTAB-Map was integrated for SLAM, fusing RGB-D camera and 3D lidar. Nav2 was configured for holonomic flight (drones strafe, ground robots don't), requiring custom transform_tolerance of 2.0s to handle RTAB-Map's continuous map updates.

Autonomous Exploration (Phase 4)

We integrated m-explore-ros2 (explore-lite) and spent significant effort debugging frontier-based exploration: BFS trapped by inflation, overly aggressive blacklisting, goal oscillation from RTAB-Map centroid shifts, and costmap resize race conditions. After extensive work, we deferred explore-lite and switched to Cosmos-driven exploration for the competition demo. The fixes are preserved in src/m-explore-ros2/ for future work.

Object Detection & Mission Control (Phases 5-6)

YOLOv8n was integrated for real-time object detection with depth-based 3D positioning, plus a terminal mission commander for natural language commands. This worked, but it was a two-system approach — YOLO detects, separate rules decide. Detection and decision-making were disconnected.

The Cosmos Pivot (Phase 7-8)

When the NVIDIA Cosmos Cookoff was announced, it made sense to unify perception and reasoning. Instead of YOLO detecting objects and a rule engine deciding what to do, Cosmos could see a scene, reason about it, and decide — all in one inference call.

First attempt: Local 2B quantized model via llama-cpp-python. The 2B suffered from decision paralysis — infinite spin loops, poor spatial reasoning from stitched 360° images, tight VRAM (3GB model + 10GB Isaac Sim on 16GB GPU).

Solution: Full 8B model on cloud GPU via vLLM. Reasoning quality improved dramatically. The 1-3s network latency became the core engineering challenge, solved by the hybrid control architecture described above.

What stayed the same: RTAB-Map SLAM, Nav2 path planning, PX4 bridge (with body→world rotation fix), TF tree, lidar-based safety.

---

Notable Technical Challenges

PX4 Velocity Frame Mismatch

PX4's TrajectorySetpoint.velocity is in world frame (ENU), not body frame. Publishing vx=0.5 (intended as "fly forward") moves the drone along a fixed world axis regardless of heading. We initially tried pre-swapping axes in individual publishers — that just changed which direction it strafed. The fix was body→world rotation using odom yaw in the PX4 bridge, applied once for all velocity sources.

Isaac Sim PX4 Y-Axis Inversion

Isaac Sim's PX4 integration has the Y-axis inverted (positive Y = South instead of North). Initially misdiagnosed as an X/Y swap. The actual fix: negate Y after body→world rotation (msg.velocity = [vx, -vy, nan]), applied in the PX4 velocity bridge.

RTAB-Map Timestamp Lag

RTAB-Map's continuous map updates caused TF timestamp drift. Nav2's default transform_tolerance: 0.1s was too strict. Setting it to 2.0s across all Nav2 nodes resolved the extrapolation errors.

Explore-Lite BFS Inflation Trapping

Frontier search BFS couldn't expand through inflated costmap cells — only ~842 out of 524K cells were at cost=0. Changing the BFS comparison from map_[nbr] <= map_[idx] to map_[nbr] < LETHAL_OBSTACLE let it traverse through inflated areas.

---

Prerequisites

• OS: Ubuntu 22.04 LTS
• ROS 2: Humble Hawksbill
• Simulator: NVIDIA Isaac Sim 5.1 (Omniverse)
• Flight Controller: PX4 SITL v1.14 + Micro-XRCE-DDS bridge
• GPU (local): NVIDIA GPU with 16GB+ VRAM (for Isaac Sim)
• GPU (cloud): NVIDIA L40S or H100 for Cosmos Reason 2 8B inference via vLLM — or set COSMOS_MOCK=true for testing without a cloud GPU

Quick Start

1. Clone and Install

git clone https://github.com/hudayfa7/sentinel-autonomous-drone.git
cd sentinel-autonomous-drone
pip install -r requirements.txt

2. Configure API Endpoint

cp .env.example .env
# Edit .env with your Cosmos Reason 2 8B endpoint:
#   COSMOS_BASE_URL=http://<nebius-ip>:8000/v1
#   COSMOS_API_KEY=<your-vllm-api-key>
#   COSMOS_MODEL=nvidia/Cosmos-Reason2-8B

For testing without a cloud GPU, set COSMOS_MOCK=true in .env for mock responses.

3. Launch Sequence

SENTINEL requires multiple terminal sessions. Launch in order:

# Terminal 1: PX4 communications bridge
MicroXRCEAgent udp4 -p 8888

# Terminal 2: Isaac Sim — open scene, press PLAY, wait ~10s

# Terminal 3: RTAB-Map SLAM (auto-waits for Isaac Sim)
bash src/auto_launch_rtabmap.sh

# Terminal 4: Static TF transforms
bash src/setup_transforms.sh

# Terminal 5: PX4 velocity bridge
python3 src/px4_velocity_bridge.py

# Terminal 6: Safety bubble (lidar collision avoidance)
python3 src/safety_bubble_node.py --ros-args -p use_sim_time:=true

# Terminal 7: Nav2 navigation stack
bash src/activate_nav2.sh

# Terminal 8: SENTINEL agent + live reasoning dashboard
# Automatically opens the Rich dashboard in a second terminal tab
source ~/ros_env/bin/activate
python3 src/cosmos_agent_node.py --ros-args -p use_sim_time:=true

# Start mission: type into the dashboard's input bar:
#   Find all survivors in the building
# Or from a separate terminal:
#   ros2 topic pub --once /cosmos/mission std_msgs/String \
#     "data: Find all survivors in the building"

---

Testing

source /opt/ros/humble/setup.bash
source ~/ros_env/bin/activate
python3 -m pytest tests/test_sentinel_components.py -v

68 tests covering: ACTION response parsing, confidence thresholds, mission memory, battery simulation, map coverage calculation, API inference, error categorization, health checks, backoff logic, compass conversion, depth projection, and self-hosted deployment configuration.

Mission Evaluation

python3 tests/evaluate_mission.py logs/sentinel_mission.log \
  --ground-truth tests/ground_truth_example.json --markdown

Post-mission evaluation: compares marked targets against ground truth positions, computes detection rate, false positive rate, API latency stats, action distribution, and anti-stuck trigger count.

Docker (Tests Only)

A Dockerfile is included for reproducible test verification. Important: this container runs the unit tests and agent code only — it does not include Isaac Sim, PX4 SITL, or the Cosmos model. The full system requires the external simulator and GPU inference setup described in Quick Start. The demo video shows the complete system in action.

docker build -t sentinel .
docker run sentinel                  # Runs 68 unit tests
docker run -it sentinel bash         # Interactive shell to explore the code

---

Evaluation

We evaluated SENTINEL across three prompt engineering approaches plus two fine-tuned models to measure how prompt design and domain-specific training affect autonomous search & rescue performance:

• v1 (Baseline): Verbose 15-line prompt with mission context, coverage %, direction recommendations, and detailed action list
• v2 (If/Then Priority): Minimal prompt with priority-ordered rules — MARK actions listed before MOVE_FORWARD
• v3 (Q&A Format): Structured yes/no questions force the model to commit to observations before choosing an action
• ft_v1 (Fine-Tuned): LoRA-adapted Cosmos Reason 2 8B trained on 106 curated pairs (84% MARK_SURVIVOR)
• ft_v2 (Balanced Fine-Tune): Retrained on 79 balanced pairs (54% MARK_SURVIVOR)

Methodology

14 total evaluation runs (plus 1 demo run) across a standardized collapsed warehouse scene.

Results

Reasoning Evolution

The same warehouse scene, same survivors — but dramatically different reasoning and behavior across prompt versions:

v1 (Baseline — Rule-Focused, Wrong Action):

<think>
The video is an industrial warehouse with two glowing green humanoid
figures. Since they're stationary and resemble mannequins, maybe they're
not survivors. The drone's task is to explore uncharted areas first.
Even though there are survivors visible, the main task is to ensure
all directions are covered.
</think>
ACTION: MOVE_FORWARD

The model sees and describes survivors but prioritizes rule-following over perception. 140+ tokens of protocol deliberation, 5% MARK rate.

v2 (If/Then — Rich Description, Still Wrong Action):

<think>
I see two green glowing human figures, which are survivors. I also
notice a pallet rack on my left and a metal handrail on my right.
I will proceed forward, making sure to avoid obstacles while keeping
track of the survivors.
</think>
ACTION: MOVE_FORWARD

The smoking gun: describes "two green glowing human figures, which are survivors" — then moves forward anyway. This is the perception-action gap.

v3 (Q&A — Correct Action, No Reasoning):

<think>yes</think>
ACTION: MARK_SURVIVOR

Forcing yes/no commitment fixed the action (30% MARK rate) but destroyed the reasoning chain. 12 tokens average.

ft_v1 (Fine-Tuned — Best Balance):

<think>
I see a green glowing humanoid figure ahead, consistent with a
survivor. Marking immediately.
</think>
ACTION: MARK_SURVIVOR

Fine-tuning preserved descriptive reasoning while fixing the action mapping. 59% MARK rate, 4.5 unique survivors/run — but also some false positives on empty corridors.

The Perception-Action Gap

The central finding across all evaluations: Cosmos Reason 2 8B can perceive survivors reliably — the <think> chains in v1 and v2 explicitly describe green mannequins as "survivors" — but the base model struggles to translate perception into the correct action. The model defaults to MOVE_FORWARD regardless of what it sees. This isn't a vision failure; it's a decision-making failure.

Fine-tuning targeted this exact gap: pairs where the image contains survivors and the correct action is MARK_SURVIVOR, paired with model outputs that say "I see survivors" but chose MOVE_FORWARD. The training data preserved the model's visual observations and only corrected the action.

---

Fine-Tuning

We fine-tuned Cosmos Reason 2 8B using LoRA (Low-Rank Adaptation) via TRL SFTTrainer — NVIDIA's recommended approach for Cosmos Reason 2.

Training Data

106 curated pairs from 9 base-model evaluation missions. 78 of 117 original pairs required correction — the model's perception text was kept intact, only the action was changed. The auto-curation pipeline (tools/curate_training_data.py) detects perception-action mismatches from the reasoning text and rewrites incorrect responses.

Training Configuration

Label Masking Bug

The first training run computed loss on ALL tokens — user prompt, image pixels, special tokens. The model was penalized for not predicting unpredictable image data, resulting in loss of 10.81 and 32% accuracy. After fixing the data collator to only compute loss on the assistant's response (<think>...</think>\nACTION: X), accuracy jumped to 84% on step 1 and reached 92.5% by epoch 5.

The Goldilocks Iteration

Two fine-tuning iterations revealed how sensitive small models are to training data distribution:

ft_v1 overcorrected — trained on 84% MARK_SURVIVOR data, it marked aggressively including on empty corridors. But it found all 3 survivors in its best run (a first), and averaged 4.5 unique survivors per run.

ft_v2 was retrained on balanced data (54% MARK) after re-curating with ft_v1's runs included. It swung back to base behavior — seeing survivors but choosing MOVE_FORWARD. The 8B model's MOVE_FORWARD prior is deeply ingrained; 54% MARK in training wasn't enough to overcome it.

The sweet spot likely sits around 70-75% MARK in the training distribution, but finding it requires more iterations than our timeline allowed. With 79-106 training pairs, small changes in action distribution cause dramatic swings in behavior.

What We Learned

1. Dataset size matters: 79-106 pairs is extremely small for vision-language fine-tuning. The model is sensitive to distribution shifts because there aren't enough examples to learn nuanced decision boundaries.
2. Single scene limitation: All training data from one warehouse layout. The model may memorize spatial patterns rather than learn generalizable behavior.
3. Label masking is critical: Without it, accuracy was 32% vs 84%. Computing loss on image/prompt tokens provides no useful gradient signal.
4. Each failure mode generates complementary training data: Base model provides correct "empty view → MOVE_FORWARD" examples; ft_v1 provides correct "survivor visible → MARK_SURVIVOR" examples. The iterative deploy → evaluate → curate → retrain cycle produces actionable insights even when individual models aren't perfect.

Fine-Tuning Pipeline

All scripts are included in tools/ for reproducibility:

# 1. Curate training data (auto-detect incorrect actions, rewrite responses)
python3 tools/curate_training_data.py

# 2. Convert to HuggingFace dataset format
python3 tools/convert_data.py

# 3. Fine-tune with LoRA (requires H100 or equivalent)
python3 tools/finetune_cosmos.py

# 4. Merge LoRA adapter into base model for vLLM deployment
python3 tools/merge_lora.py

---

Project Structure

sentinel-autonomous-drone/
├── .env.example                    # API configuration template
├── requirements.txt                # Python dependencies
├── Dockerfile                      # Reproducible test container
├── src/
│   ├── cosmos_agent_node.py        # SENTINEL agent — Cosmos Reason 2 8B (~2200 lines)
│   ├── sentinel_display.py         # Rich terminal dashboard
│   ├── safety_bubble_node.py       # Lidar collision avoidance (20Hz)
│   ├── depth_projection.py         # Pixel-to-3D projection utility
│   ├── nav2_drone_params.yaml      # Nav2 configuration for holonomic drones
│   ├── px4_velocity_bridge.py      # PX4 bridge (body→world rotation + axis swap)
│   ├── activate_nav2.sh            # Nav2 launch + lifecycle activation
│   ├── setup_transforms.sh         # TF tree static publishers
│   ├── auto_launch_rtabmap.sh      # RTAB-Map with Isaac Sim readiness wait
│   ├── wait_for_isaac.sh           # Sensor availability checker
│   ├── yolo_semantic_node.py       # (legacy) YOLOv8 detection — superseded by Cosmos
│   ├── coverage_mission.py         # (legacy) Waypoint coverage patterns
│   ├── mission_commander.py        # (legacy) Natural language command parser
│   ├── smart_explore.py            # (legacy) Autonomous exploration starter
│   └── m-explore-ros2/             # (legacy) Modified explore-lite source
├── tools/
│   ├── curate_training_data.py     # Auto-detect incorrect actions, rewrite responses
│   ├── finetune_cosmos.py          # LoRA fine-tuning via TRL SFTTrainer
│   ├── merge_lora.py               # Merge LoRA adapter into base model for vLLM
│   └── convert_data.py             # LLaVA JSON → HuggingFace Dataset converter
├── tests/
│   ├── test_sentinel_components.py # 68 unit tests
│   ├── evaluate_mission.py         # Post-mission metrics evaluation
│   ├── ground_truth_warehouse.json # Demo warehouse ground truth (3 survivors, 2 hazards)
│   ├── ground_truth_example.json   # Sample ground truth for evaluation
│   └── E2E_CHECKLIST.md            # Manual testing checklist
├── demo/
│   ├── README.md                   # Demo run summary and log descriptions
│   ├── mission_log.csv             # Best run telemetry (3/3 survivors, 0 false positives)
│   ├── reasoning_log.txt           # Full Cosmos reasoning chains
│   └── object_db.json              # Detected targets with 3D positions
└── logs/                           # All evaluation run logs (14 runs + demo)
    ├── base_v1/                    # Prompt v1 baseline (3 runs)
    ├── base_v2/                    # Prompt v2 if/then (3 runs)
    ├── base_v3/                    # Prompt v3 Q&A (3 runs)
    ├── ft_v1/                      # Fine-tuned v1 (4 runs)
    ├── ft_v2/                      # Fine-tuned v2 (1 run)
    └── demo_best/                  # Best run — 3/3 survivors, 0 false positives

---

Known Limitations

Hazard Detection Gap

Across all 14 evaluation runs (9 base model + 5 fine-tuned), the model detected zero hazards. Our warehouse scene uses bright orange/red emissive cones as fire stand-ins, but Cosmos Reason 2 consistently describes them as "yellow objects" or "wooden pallets" rather than recognizing them as hazards. The model has never seen these specific objects labeled as fire — and real fire looks nothing like a glowing cone. This is a training data gap, not a model failure: with real fire footage or Omniverse Flow fire effects in the training set, the model would likely learn the association. We couldn't use Omniverse Flow fire in our scene because it exceeded our GPU's 16GB VRAM (RTX 5070 Ti) — Isaac Sim alone uses 13+ GB, leaving no room for volumetric fire simulation. A 24GB+ GPU (RTX 4090, A6000) would resolve this. In real-world deployment, a thermal camera would bypass this limitation entirely — fire is unmistakable in infrared regardless of visual appearance.

Single-Scene Evaluation

All evaluation runs use the same warehouse scene with fixed survivor/hazard positions. While we vary the model and prompt version across runs, the spatial layout is constant. This means our metrics reflect performance on one specific environment, not generalized S&R capability. More diverse scenes (multi-room, multi-floor, varied lighting, different survivor poses) would provide stronger evidence of robustness.

Cloud Inference Latency

The 1-3 second API round-trip to the Nebius cloud GPU means the drone hovers for 1-3 seconds between every decision. In a real collapse scenario with active fire spread or aftershocks, this latency could be critical. Edge deployment on Jetson Orin with a quantized model is the path to sub-second inference.

---

Future Work

SENTINEL was built for a competition, but the problem it addresses is real. 89 firefighters died in the line of duty in 2023, with structural collapse among the leading causes. The survival rate for trapped victims drops from 90% to under 30% within 72 hours. A drone that can fly ahead, map the space, and report survivor locations before anyone enters changes the risk equation for every structural collapse response.

Phase 1: Edge Deployment & GPS-Denied Operation

The current system relies on a cloud GPU for Cosmos inference. For real deployment, the model runs on edge hardware (Jetson Orin) with no network dependency. Collapsed buildings and underground spaces have no GPS or cellular coverage — the system must be entirely self-contained. Our hybrid architecture was designed for this: the local safety bubble already runs at 20Hz on-device. Quantization (INT8/INT4) and model distillation target inference under 5 seconds on Jetson.

Phase 2: Real Hardware with Thermal Imaging

The physical platform: custom quadcopter with Pixhawk 6C, Jetson Orin Nano, Intel RealSense D435 (RGB-D), and a FLIR thermal camera. Thermal imaging is essential — detecting body heat through smoke, dust, and darkness where RGB cameras fail. In simulation we approximate this with green-emissive mannequins; real deployment needs real thermal sensors.

Phase 3: Scaled Fine-Tuning for Disaster Scenarios

The base Cosmos Reason 2 8B understands physical scenes, but S&R has edge cases general training doesn't cover: partially buried survivors, structural instability indicators, gas leak signs, electrical hazards. Fine-tuning on hundreds of disaster-specific scenarios (fire, flood, earthquake, structural collapse) would improve reliability and reduce false positives. Our evaluation framework already collects training pairs automatically — every mission generates labeled data.

Phase 4: Multi-Drone Coordination

Real disaster sites need multiple drones working together. This connects with our parallel work on autonomous swarm coordination, where drones share maps, divide search areas, and avoid duplicating effort. Research shows five autonomous UAVs can cover 2 km² in under 90 minutes with 90% coverage — a capability that multiplies with coordination.

Phase 5: NVIDIA Cosmos Predict

NVIDIA Cosmos Predict generates physically accurate future world states. Integrating Predict alongside Reason 2 would enable the drone to anticipate structural changes (collapsing debris, spreading fire) and plan paths that account for predicted hazards — moving from reactive to predictive autonomy.

Phase 6: Survivor Triage & Two-Way Communication

Finding survivors is step one. The next step is triage — a drone equipped with a speaker and microphone can communicate with conscious survivors, assess severity via VLM reasoning (trapped vs. mobile), and generate triage classifications (RED/YELLOW/GREEN) for incident commanders. The <think> reasoning already analyzes the scene — extending it to assess survivor condition is a natural evolution.

Phase 7: Multi-Model Pipeline (Perceive → Reason → Act)

The current system asks a single 8B model to do everything in one pass. Our testing revealed this overloads the model — it can identify survivors but selects the wrong action. A production system would decompose into: Perceive (lightweight detector), Reason (Cosmos Reason 2 with structured input), Act (policy model or rule engine). Each model operates within its strength.

Phase 8: Full Autonomous Navigation + Separate VLM Reasoning

Currently, SENTINEL relies on Cosmos for both visual understanding AND navigation decisions. The proper architecture separates these: a navigation layer (frontier exploration, coverage planning) decides where to go, while a VLM reasoning layer (Cosmos) decides what it sees. This solves the turn-loop and coverage-gap issues that pure VLM navigation produces.

Phase 9: Confidence-Based Target Verification

Currently, SENTINEL marks targets at a single threshold. A more robust approach would use confidence levels: high confidence (>70%) marks immediately, medium (40-70%) moves closer for verification, low (<40%) logs as possible and revisits later. The <think> chain already contains confidence signals that could be parsed into adaptive behavior.

Phase 10: Integration with Emergency Response

The end goal is a tool for existing first responder workflows: survivor locations in formats incident commanders use, integration with dispatch and GIS systems, and a real-time standalone dashboard for non-technical operators. The <think> reasoning chain is valuable here — responders see why the drone flagged a location, not just that it did.

---

Team

Team Sentinel | Loopworks NVIDIA Cosmos Cookoff 2026

License

Apache License 2.0