MultiPhysio-HRC: Multimodal Physiological Signals Dataset for Industrial Human-Robot Collaboration

Companion GitHub repository for the paper “MultiPhysio-HRC: Multimodal Physiological Signals Dataset for Industrial Human-Robot Collaboration.”
This repo hosts code, preprocessing pipelines, feature extraction, and baseline models to reproduce results from the paper.

---

📖 Overview

MultiPhysio-HRC is a multimodal dataset and toolkit for mental-state estimation in industrial Human-Robot Collaboration (HRC).

This repository provides:

• Loaders and preprocessing for physiological, EEG, voice, and AU data.
• Feature extraction pipelines for all modalities.
• Baseline regression and classification models with LOSO-CV evaluation.
• Example notebooks to explore the dataset and reproduce paper results.

---

📂 Repository Structure

.
├─ paper/                          # Camera‑ready or preprint PDF, figures (optional)
├─ docs/
│  ├─ dataset_overview.md          # Modalities, tasks, questionnaires, ethics
│  ├─ data_schema.md               # File formats, splits, naming, timestamps
│  └─ benchmarks.md                # Baseline setups & expected metrics
├─ src/
│  ├─ dataprep/                    # Loading, syncing, cleaning
│  ├─ features/                    # Physio, EEG, voice, AUs feature extraction
│  ├─ models/                      # Baselines (RF/AB/XGB), utils
│  └─ eval/                        # Metrics, LOSO CV, reporting
├─ notebooks/
│  ├─ 01_quicklook.ipynb           # Explore a subject & modalities
│  ├─ 02_extract_features.ipynb    # End‑to‑end feature extraction
│  └─ 03_train_baselines.ipynb     # Reproduce results from the paper
├─ examples/
│  └─ minimal_pipeline.py          # Scripted end‑to‑end run
├─ requirements.txt                # Python deps
└─ README.md                       # You are here

Tip: If you keep raw data outside the repo, set MULTIPHYSIO_HRC_DATA env var to the dataset root to avoid passing paths around.

---

Getting Started

1) Install

# create a clean env (conda or mamba recommended)
conda create -n mphrc python=3.10 -y
conda activate mphrc

# install dependencies
pip install -r requirements.txt

2) Download the dataset

• Visit https://zenodo.org/records/17225571 and follow the instructions to obtain access and download files.
• Keep the raw data in a folder of your choice and set:

export MULTIPHYSIO_HRC_DATA=/path/to/MultiPhysio-HRC

3) Sanity‑check a subject

jupyter lab  # then open notebooks/01_quicklook.ipynb

---

Data Schema

MultiPhysio-HRC/
│
├── physiological_data/
│   ├── filtered/                # Preprocessed signals
│   │   ├── subj1/
│   │   │   ├── task1.csv
│   │   │   ├── task2.csv
│   │   │   ...
│   │   └── subj2/
│   │       ├── task1.csv
│   │       ├── task2.csv
│   │       ...
│   │
│   └── raw/                     # Raw signals as acquired
│       ├── subj1/
│       │   ├── task1.csv
│       │   ├── task2.csv
│       │   ...
│       └── subj2/
│           ├── task1.csv
│           ├── task2.csv
│           ...
│
├── features/                    # Extracted features and labels
│   ├── aus_data.csv
│   ├── bio_features_60s.csv
│   ├── eeg_features_5s.csv
│   ├── nlp_embeddings.csv
│   ├── speech_features.csv
│   └── labels.csv
|
└── README.md

• Windows: Physio features on 60 s windows; EEG on 5 s windows; AUs at 2 fps.

---

Reproducing the Paper Baselines

End‑to‑end (script):

python examples/minimal_pipeline.py \
  --data $MULTIPHYSIO_HRC_DATA \
  --modality physio \
  --task regression --label STAI \
  --cv loso --report out/report_physio_stai.json

Notebooks:

1. 02_extract_features.ipynb – computes features for Physio/EEG/Voice/AUs.
2. 03_train_baselines.ipynb – trains RF / AdaBoost / XGBoost for regression & 3‑class classification (Low/Med/High) based on per‑subject z‑like thresholds.

Models: RandomForest, AdaBoost, XGBoost. Evaluation uses Leave‑One‑Subject‑Out (LOSO). Features & labels are min–max normalized within‑subject as in the paper.

---

Results (from the paper)

• Regression (STAI‑Y1 & NASA‑TLX): Physiological features yield the lowest RMSE, stronger than EEG and Voice.
• 3‑Class Classification (Stress & Cognitive Load): Physiological features achieve the highest F1, with EEG close behind for cognitive load; Voice trails Physio/EEG.

See docs/benchmarks.md for expected ranges and how we compute the Low/Med/High bins per subject.

---

FAQ

Q: How do I get access to raw videos or robot logs?
A: See the dataset page. Some assets may require additional request/agreements.

Q: Are there ready‑made splits?
A: We default to Leave‑One‑Subject‑Out. Utility functions can generate stratified splits by task/condition.

---

Citation

If you use MultiPhysio‑HRC or this code, please cite the paper:

@article{robotics14120184,
      AUTHOR = {Bussolan, Andrea and Baraldo, Stefano and Avram, Oliver and Urcola, Pablo and Montesano, Luis and Gambardella, Luca Maria and Valente, Anna},
      TITLE = {MultiPhysio-HRC: A Multimodal Physiological Signals Dataset for Industrial Human-Robot Collaboration},
      JOURNAL = {Robotics},
      VOLUME = {14},
      YEAR = {2025},
      NUMBER = {12},
      ARTICLE-NUMBER = {184},
      URL = {https://www.mdpi.com/2218-6581/14/12/184},
      ISSN = {2218-6581},
      DOI = {10.3390/robotics14120184}
}

---

Acknowledgments & Funding

• Horizon Europe — FLUENTLY (Grant 101058680)
• Eurostars — !2309‑Singularity
• We thank all participants and the technical staff who supported the acquisition campaign.

---

Ethics & License

This dataset was collected under institutional ethical approval (SUPSI), with informed consent from all participants.

• Code: Licensed under the MIT License.
• Dataset: Released under CC BY 4.0.

---

Contact

• Lead contact: andrea.bussolan@supsi.ch
• Issues & questions: please open a GitHub issue.

---

Maintainers: Andrea Bussolan, Stefano Baraldo.