Continuous Sign Language Recognition (CSLR)

Overview

This repository contains experimental research on Continuous Sign Language Recognition (CSLR). The objective of the project was to explore architectural complexity limits by integrating advanced temporal and spatial modeling techniques. The resulting architecture achieved a Word Error Rate (WER) of 15.7%, which significantly outperformed the 60% Transformer baseline.

The approach:

• Spatial Dependencies: Architected a Graph Neural Network (GNN) solution that models the human skeleton as a graph structure to capture spatial dependencies in sign language.
• Temporal Dynamics: Integrated Temporal Convolutional Networks (TCNs) and Transformers (via an ensemble of GNN variants combined with a GNN-BiLSTM hybrid) to capture temporal dynamics.

Results and Takeaways

This project was purely for experimenting and learning. The main GNN architecture achieved a Word Error Rate (WER) of 15.7%, which significantly outperformed our custom-built Transformer experiment (ID-CNN + AraGPT2) which yielded a 60% WER.

However, the GNN model underfitted the data due to severe compute limits. This was an important lesson in architectural design: ensembling multiple Graph Neural Networks drastically increases the parameter count and the compute required to train them effectively. The bottleneck was a direct result of underestimating the true computational cost that ensembling entails.

Credits and Inspiration

This implementation builds upon concepts and code from two main sources:

1. Base Framework: MSLR_ICCV2025 by VIPL-SLP. Their repository served as the backbone for the GNN experiment. All credit for the base framework goes to the original authors.
2. Ensemble Concept: The GNN ensemble approach was inspired by the paper Region-Aware Pose Modeling and Permutation Decoding for Signer-Independent Sign Language Recognition.

Note: The TCN addition was an experimental "sprinkle" introduced to observe how Temporal Convolutional Networks could further refine the temporal modeling.

Repository Structure

• train_gnn.py: Setup script that clones the base MSLR repository and automatically injects the custom GNN/TCN ensemble modifications for training.
• train_transformer.py: Standalone script containing the custom ID-CNN + AraGPT2 transformer experiment, built specifically to serve as a comparative baseline.
• src/: Core Python modules (the modifications).
  • slr_network.py: The main recognition network combining visual extraction with temporal sequences.
  • stgcn_block.py: High-speed Spatial-Temporal Graph Convolutional Network blocks including TurboGAT and GNN Ensemble.
  • seq_scripts.py: Custom training and evaluation sequences.
• notebooks/: Jupyter notebooks used for implementation and training via Colab.
• assets/: Relevant figures and training logs.