SignSparK-BT

Back-translation for sign language production: takes SMPL+MANO pose features and produces spoken language text. Use it to score generated sign sequences by translating the poses back to text and comparing it against the source sentence.

This is the evaluation companion to SignSparK and it reads from the same LMDB datasets.

---

Quickstart

From a clean checkout to a score, in four steps.

# 1. install
conda env create -f environment.yml && conda activate signspark-bt
pip install -e .

# 2. download pose data (from the SignSparK repo) and point it to the download location
python tools/download_data.py --datasets PHOENIX-2014T --dest ./data
export DATA_ROOT=$(pwd)/data

# 3. download our back-translation models
python tools/download_models.py --datasets PHOENIX-2014T --dest ./checkpoints

# 4. check it works: this should print BLEU-4 15.00
signspark-bt score checkpoints/PHOENIX-2014T --split test

Please make sure you have two things pointed at the right place:

DATA_ROOT is only needed for train, test, and score --split. Scoring a sample dump or your own features needs the model directory alone.

---

Install

Conda, for a pinned and reproducible environment:

conda env create -f environment.yml
conda activate signspark-bt

Or pip:

pip install -r requirements.txt   # pinned, equivalent to environment.yml
pip install -e .

Our provided requirements are targeted for CUDA 12.8 wheels, needed for Blackwell / RTX 50-series. For a different CUDA version change the index URL and drop the +cu128 tag.

To install just the package and let pip resolve loose dependency bounds:

pip install -e ".[zh,logging]"

Requires Python 3.9+, PyTorch and LMDB.

Data

The pose data is distributed by SignSparK on the HuggingFace Hub at LionelLow/SignSparK_data:

python tools/download_data.py --datasets CSL-Daily How2Sign PHOENIX-2014T --dest ./data
export DATA_ROOT=$(pwd)/data

(tools/download_data.py lives in the SignSparK repo.) The expected layout is:

$DATA_ROOT/lmdb/{train,dev,test}/<Dataset>_reopt_<split>.lmdb

Each clip provides a translation, a gloss, and per-frame SMPL-X 6D rotations. Our backtranslation model consumes a 240-dimensional vector per frame:

Copy .env.example to .env and fill in DATA_ROOT and, if you use W&B, WANDB_API_KEY and WANDB_ENTITY.

Pseudo-glosses for How2Sign

As How2Sign is not gloss annotated, we generate pseudo-glosses as a naive fix (detailed in supplementary):

python -m spacy download en_core_web_lg   # 3.8.0
python tools/make_pseudo_glosses.py --dataset How2Sign

Output is sensitive to the spaCy model version, so pin en_core_web_lg==3.8.0 if you need to reproduce a specific run.

Usage

# Train
signspark-bt train configs/phoenix14t.yaml

# Evaluate a checkpoint on the dev and test splits
signspark-bt test configs/phoenix14t.yaml --ckpt checkpoints/bt_phoenix14t/best.ckpt

Training writes to the config's model_dir (checkpoints/bt_phoenix14t), while downloaded models land in checkpoints/<Dataset> (checkpoints/PHOENIX-2014T). Both layouts work anywhere a model directory is expected.

Any config value can be overridden on the command line:

signspark-bt train configs/how2sign.yaml --training.batch_size 64 --training.learning_rate 1e-4

The first run indexes every clip in the LMDB, which is I/O bound. The index is cached under $DATA_ROOT/.bt_index_cache, so later runs start immediately.

Scoring sign language production

Additionally, we provide a score script to easily back-translate poses and report BLEU, chrF and ROUGE. Given a SignSparK sample .npy dump, the script scores the ground-truth and the generated poses in one pass and reports the drop between them:

signspark-bt score checkpoints/<model> \
    --body-npy  .../body/<run>/seed102_clampstep0_<dataset>.npy \
    --hand-npy  .../hand/<run>/seed102_clampstep0_<dataset>.npy

CSL-Daily needs the zh extra: its text is segmented with jieba and scored per character, so its chrF is not comparable with the other two.

Note that the ground-truth values are the upper-bound of the back-translation model. The body and hand dumps should come from the same sample_all.py run in the SignSparK repo. Also note that the face stream is unused: the model only takes body (60) plus both hands (90 each).

Note: Scores reported here differ from the main paper. Both the ground-truth and predicted pose scores here have been recomputed using a back-translation model retrained to match the reoptimized LMDBs released at LionelLow/SignSparK_data. The back-translation drop is also smaller than in the paper, since the released checkpoints are trained on 15× more data of higher quality.

Using a trained model from Python

import torch
from signspark_bt import make_back_translation_model, back_translate

model = make_back_translation_model("checkpoints/PHOENIX-2014T")

# B x T x 240 SMPL+MANO features: body (60) + left hand (90) + right hand (90)
poses = torch.rand(8, 64, 240)

sentences = back_translate(model=model, poses=poses)

poses may also be a list of T x 240 tensors of differing length. Empty or None entries come back as empty strings at their original positions, so the output always lines up with the input. Beam size and alpha are read from each model's config.yaml, so you do not need to set them by hand.

Download the Pre-trained models

The back-translation models are on the HuggingFace Hub at LionelLow/SignSparK_BT:

python tools/download_models.py --datasets PHOENIX-2014T CSL-Daily How2Sign --dest ./checkpoints

After download, each dataset directory should hold config.yaml, best.ckpt and two vocabulary files. See Scoring sign language production to find our updated back translation scores.

Note: Our original back-translation scores in the paper was trained on the raw, unoptimization data. This has since been superseded by this official release to match our improved data extractions.

Configuration

Configs live in configs/. ${VAR} and ~ are expanded anywhere in the file.

The fields most worth knowing:

Tests

pip install -e ".[dev]"
pytest

The suite covers CTC decoding, collation and masking, vocabularies, metrics and pseudo-gloss generation. It needs no data and no GPU.

Citation

If you use this code, the released back-translation models, or the SignSparK data, please cite:

@inproceedings{low2026signspark,
  title={SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learning},
  author={Low, Jianhe and Symeonidis-Herzig, Alexandre and Ivashechkin, Maksym and Sincan, Ozge Mercanoglu and Bowden, Richard},
  booktitle={European Conference on Computer Vision},
  pages={648--670},
  year={2026},
  organization={Springer}
}

Licence

Apache 2.0; see LICENSE. This code derives from JoeyNMT and Sign Language Transformers, and vendors corpus BLEU and chrF from sacreBLEU and ROUGE-L from coco-caption. We thank them for open-sourcing their code and making this codebase possible.

The underlying datasets (CSL-Daily, How2Sign, RWTH-PHOENIX-Weather 2014T) carry their own licences, which govern any use of the data and of models trained on it.