🤖 P.E.P.Pe — Personal Enhanced Planner with PEpper

A ROS2-based cognitive robotic assistant running on the Pepper humanoid robot that manages personal To-Do Lists through natural spoken language. The system integrates face-based user identification, LLM-powered dialog management (Mistral via Azure AI), automatic speech recognition (Whisper), and a real-time web dashboard on Pepper's tablet.

📄 Developed as a university project at Università degli Studi di Salerno — DIEM (Cognitive Robotics course), Group 02.

---

📋 Table of Contents

• Project Overview
• System Architecture
• ROS2 Node Graph
• Project Structure
• Requirements
• Installation
• Configuration
• Running the System
• Running in Simulation (No Robot)
• NLU Pipeline & Dialog Management
• User Management & Persistence
• Test Cases

---

🔍 Project Overview

P.E.P.Pe turns the Pepper robot into a personal task-management assistant. The user interacts entirely through natural spoken Italian, and the system:

• Identifies the user by face (biometric embedding, no credentials)
• Understands free-form requests via an LLM with Tool Calling
• Manages a persistent per-user To-Do List (add, remove, read, modify tasks)
• Responds via Pepper's speakers (TTS)
• Displays the live task list and a video feed on Pepper's chest tablet

Operational Constraints

---

🏗️ System Architecture

Node Responsibilities

---

🗺️ ROS2 Node Graph

Topics

Services

---

📁 Project Structure

P.E.P.Pe-Personal_Enhanced_Planner_with_PEpper/
│
└── project_ws/                        # ROS2 workspace
    └── src/
        │
        ├── PEPPe/                     # Main application package
        │   ├── PEPPe/
        │   │   ├── LINo.py            # LLM Interaction Node (orchestrator)
        │   │   ├── GINo.py            # Graphic Interface Node (tablet web server)
        │   │   ├── PINo.py            # People Identification Node (face ID)
        │   │   ├── fake_od.py         # Stub: fake object detection (simulation)
        │   │   ├── fake_stt.py        # Stub: fake speech-to-text (simulation)
        │   │   ├── fake_tts.py        # Stub: fake text-to-speech (simulation)
        │   │   ├── requirements.txt   # Python dependencies for this package
        │   │   └── users_list/        # User database (JSON persistence)
        │   ├── launch/                # ROS2 launch files
        │   ├── setup.py
        │   ├── setup.cfg
        │   └── package.xml
        │
        ├── pepper_nodes/              # Pepper hardware interface package
        │   ├── pepper_nodes/          # ReSpeaker (RINo), ASR, TTS nodes
        │   ├── conf/                  # ROS2 parameter YAML files
        │   ├── launch/                # Launch files for hardware nodes
        │   ├── setup.py
        │   └── package.xml
        │
        ├── pepper_interfaces/         # Custom ROS2 message/service definitions
        │
        └── audio_common/              # Audio utilities / common audio ROS2 package

---

⚙️ Requirements

• ROS2 Humble or later
• Python 3.10+
• Pepper robot (NAOqi 2.9 compatible) — or use the simulation stubs (fake_*.py)
• Azure AI Inference credentials (for Mistral-medium LLM access)
• ReSpeaker v2.0 USB microphone array (or standard mic for simulation)

Key Python Dependencies

See project_ws/src/PEPPe/PEPPe/requirements.txt for the full pinned list.

---

🛠️ Installation

# 1. Clone the repository
git clone https://github.com/antocarbone/P.E.P.Pe-Personal_Enhanced_Planner_with_PEpper.git
cd P.E.P.Pe-Personal_Enhanced_Planner_with_PEpper

# 2. Source your ROS2 installation
source /opt/ros/humble/setup.bash

# 3. Install Python dependencies
pip install -r project_ws/src/PEPPe/PEPPe/requirements.txt

# 4. Build the ROS2 workspace
cd project_ws
colcon build --symlink-install

# 5. Source the workspace overlay
source install/setup.bash

---

🔧 Configuration

Before launching, set the following environment variables or edit the relevant YAML parameter files in project_ws/src/pepper_nodes/conf/:

# Azure AI credentials for the LLM
export AZURE_INFERENCE_ENDPOINT="https://<your-endpoint>.inference.ai.azure.com"
export AZURE_INFERENCE_KEY="<your-api-key>"

# Pepper robot IP (if running on real hardware)
export PEPPER_IP="<robot-ip>"

ROS2 Parameters (pepper_nodes/conf/)

---

🚀 Running the System

Full System on Real Pepper

# Terminal 1 — Hardware nodes (ReSpeaker, ASR, TTS, camera)
ros2 launch pepper_nodes pepper_hardware.launch.py pepper_ip:=<ROBOT_IP>

# Terminal 2 — Main application nodes (LINo, GINo, PINo)
ros2 launch PEPPe peppers_application.launch.py

Once running, Pepper's tablet will automatically open http://<ROBOT_IP>:8000 showing the live dashboard.

---

🧪 Running in Simulation (No Robot)

The fake_*.py stubs allow full end-to-end testing without a physical Pepper robot:

# Use fake STT (type text instead of speaking)
ros2 run PEPPe fake_stt

# Use fake TTS (prints to console instead of speaking)
ros2 run PEPPe fake_tts

# Use fake object detection (publishes a dummy embedding)
ros2 run PEPPe fake_od

# Then launch the core logic nodes normally
ros2 launch PEPPe peppers_application.launch.py

---

🧠 NLU Pipeline & Dialog Management

The cognitive core is LINo, which uses Mistral-medium via Azure AI Inference with the Tool Calling paradigm.

Available LLM Tools

Prompt Engineering

Every user message is enriched with metadata before being sent to the LLM:

timestamp: YYYY-MM-DD HH:MM:SS | nome: <USERNAME> | <USER_MESSAGE>

This allows the model to resolve relative time expressions (e.g., "domani alle sedici e mezza" → date: 2026-01-30, time: 16:30) without maintaining internal state.

Ego-Speech Suppression

To prevent the robot from transcribing its own voice in an infinite loop:

1. Pre-TTS: LINo calls mic_active(False) → RINo disables VAD and flushes audio buffers
2. TTS speaks: Pepper's voice plays out loud
3. Post-TTS callback: LINo calls mic_active(True) → listening resumes

---

👤 User Management & Persistence

User data is stored in users_list/users_db.json. Each record contains:

{
  "id": 1,
  "nome": "Franco",
  "embedding": [0.23, -0.11, ...],
  "todo_list": {
    "fare la spesa": "2026-01-28T21:00:00",
    "andare dal commercialista": "2026-01-30T16:30:00"
  }
}

Re-identification Logic

• On each frame, PINo publishes a 512-dim face embedding on /scene_embedding
• LINo computes Euclidean distance to all stored embeddings
• If distance < embedding_threshold → user recognised, context restored
• If distance ≥ threshold → new user, system asks for their name and calls register_user
• Stability guard: a new identity must be confirmed for PENDING_TIME seconds before a context switch triggers, preventing flicker from brief misdetections

---

✅ Test Cases

The system was validated with 9 black-box test cases — all passed: