Fine-tuned RoBERTuito for Spanish Emotion Classification

This project fine-tunes RoBERTuito, a RoBERTa model pretrained on Spanish, to classify short autobiographical texts into six emotions: happiness, sadness, disgust, anger, fear and surprise. It was developed as an NLP research project at BUAP (Benemérita Universidad Autónoma de Puebla).

Why emotion classification in Spanish

Most emotion classification research and tooling targets English. Spanish is spoken by more than 480 million people, and models trained on English text transfer poorly to Spanish because of differences in syntax, informal register, and culturally specific ways of expressing emotion. Reliable emotion classifiers in Spanish are useful for mental health screening tools, social media monitoring, and customer feedback analysis in Spanish-speaking markets, none of which are well served by translating text into English first.

Why RoBERTuito instead of a multilingual model

RoBERTuito is a RoBERTa-based model pretrained from scratch on approximately 500 million Spanish tweets, rather than being one of ~100 languages in a shared vocabulary like XLM-R or multilingual BERT. Because its entire pretraining budget and vocabulary are spent on Spanish (including the informal, colloquial register typical of user-generated text), it starts from a better representation of Spanish than a multilingual model would for this kind of short, informal, first-person text. This project uses pysentimiento/robertuito-base-uncased.

Dataset

The dataset (data/emotion_survey.csv) is a custom survey, not a public benchmark. It was collected via a Google Form from 341 respondents, mostly students at the Facultad de Ciencias de la Computación (BUAP). Each respondent answered 12 open-ended prompts, two per emotion (e.g. "Recuerda un momento en el que lograste algo que realmente deseabas" for happiness, "Recuerda un día en el que sentiste que todo te salía mal" for sadness). Respondent IDs are anonymized, but the free-text answers are personal, self-reported narratives.

Labels: Felicidad (happiness), Tristeza (sadness), Disgusto (disgust), Ira (anger), Miedo (fear), Sorpresa (surprise).

Before filtering, each emotion has close to the same number of raw answers (Felicidad 682, Tristeza 682, Disgusto 681, Ira 682, Miedo 682, Sorpresa 682), so the dataset is approximately balanced by construction — every respondent answers two prompts per emotion. After removing empty, very short (fewer than 3 words), or boilerplate answers ("no", "nada", "no recuerdo", etc.), 1,860 examples remain, split 1,488 / 372 (80% / 20%, stratified, seed 42).

The dataset is included in this repository for reproducibility only. It contains personal narrative data from real survey participants and is not covered by the MIT license — see Limitations and the note in LICENSE.

Methodology

Two pipelines were explored:

1. TF-IDF + classical classifiers (notebooks/emotion_classification_robertuito.ipynb, first half): grammar correction (language_tool_python), text normalization, spaCy tokenization and lemmatization, TF-IDF vectorization, then SVM / KNN / Naive Bayes / decision trees / Random Forest. This serves as a non-transformer baseline.
2. RoBERTuito fine-tuning (second half of the same notebook, and src/train.py): raw text is tokenized directly with the RoBERTuito tokenizer (max_length=128, which covers the large majority of answers without truncation) and fed to RobertaForSequenceClassification via Hugging Face Trainer.

Fine-tuning hyperparameters, exactly as used:

A small, unfinished grid search over learning rate, epochs, batch size and weight decay (itertools.product over 16 combinations) was also run; of the 6 combinations that completed before the search was interrupted, the best reached 85.2% accuracy / 0.850 F1 macro (lr=2e-5, epochs=12, batch_size=8, weight_decay=0.01). This was not run to completion and is not the headline result.

Results

Three setups were evaluated on the same 372-example held-out test set:

• Pretrained RoBERTuito + SVM (no fine-tuning): 75.8% accuracy. Sentence embeddings from the off-the-shelf model, without any task-specific training, fed into an SVM.
• Fine-tuned RoBERTuito classifier head: 83.3% accuracy, 0.833 F1 macro, 0.833 recall macro, 0.834 precision macro. This is the direct, end-to-end result of RobertaForSequenceClassification fine-tuned with Trainer, and the number to cite as "the fine-tuned transformer's" own performance.
• Fine-tuned RoBERTuito embeddings + SVM: 96.0% accuracy, 0.96 F1 macro. Here RoBERTuito is used purely as a feature extractor after fine-tuning, and a separate SVM is trained on its CLS embeddings. This two-stage setup was also validated with 5-fold and 10-fold stratified cross-validation, giving 94.4% (± 1.3%) and 94.6% (± 1.8%) average accuracy respectively, so the ~96% single-split result is not a lucky split.

These are two different architectures, not two measurements of the same model, and both numbers are legitimate — see Limitations for a caveat that applies to the second one.

Training curves (fine-tuned classifier head):

Confusion matrix, fine-tuned classifier head (83.3% accuracy):

Confusion matrix, SVM on fine-tuned embeddings (96.0% accuracy):

Per-class precision / recall for the SVM-on-embeddings setup (62 test examples per class):

Class distribution in the raw survey data:

Limitations

• Small dataset. 1,860 examples from 341 respondents is small for fine-tuning a transformer; results can be sensitive to the specific train/test split.
• Respondent-level leakage risk. Each respondent contributes one example per emotion (up to 6 examples total), and the train/test split is done at the example level, not grouped by respondent. The same person's writing style or vocabulary can therefore appear in both train and test, which can inflate metrics relative to a split that fully separates respondents.
• Test set reused for model selection and final evaluation. load_best_model_at_end picks the best epoch using the same 372-example validation set that is then reported as the final result, and that same set is reused again to evaluate the SVM trained on fine-tuned embeddings. This is common practice with small datasets but is not the same as evaluating on a fully independent held-out set, and likely inflates the reported numbers somewhat.
• Narrow domain. The text is first-person, autobiographical survey answers to fixed prompts, mostly from BUAP students in their early twenties. Performance is not expected to transfer directly to other domains such as social media posts, product reviews, or dialogue.
• Register mismatch. RoBERTuito was pretrained on Twitter Spanish, a short and informal register. Survey answers are longer and more narrative, so there is a domain gap between pretraining and fine-tuning data, even though fine-tuning still improves results substantially.
• Dataset consent and licensing. The dataset contains real, personal narrative responses. It is included here for reproducibility of this project only, is not covered by the MIT license, and should not be redistributed or reused without separately verifying consent.

Reproducing the training

git clone https://github.com/ALNM7/fine-tuned-transformers-emotions.git
cd fine-tuned-transformers-emotions

pip install -r requirements.txt
python -m spacy download es_core_news_sm

To reproduce the fine-tuned classifier head end to end:

cd src
python train.py --csv-path ../data/emotion_survey.csv --output-dir robertuito_emotion_model

To run inference with a trained model:

python src/predict.py --model-dir src/robertuito_emotion_model "Que agradable sorpresa el que hayas venido"

The full exploratory analysis, the TF-IDF baseline, and the fine-tuning + evaluation code with their original outputs are in notebooks/emotion_classification_robertuito.ipynb.

Repository structure

fine-tuned-transformers-emotions/
├── notebooks/
│   └── emotion_classification_robertuito.ipynb   exploratory analysis, TF-IDF baseline, RoBERTuito fine-tuning and evaluation
├── src/
│   ├── data.py      dataset loading and text filtering shared by train.py
│   ├── train.py     standalone script that reproduces the fine-tuning run
│   └── predict.py   inference on a saved fine-tuned model
├── figures/         exported evaluation figures embedded in this README
├── data/
│   └── emotion_survey.csv   raw survey data (see Limitations for usage restrictions)
├── requirements.txt
└── LICENSE          MIT, code only (see note in the file)