Multilingual Sentiment (base, 308M): negative / neutral / positive

A multilingual sentiment classifier built on mmBERT-base. It returns negative, neutral or positive for reviews, social posts, comments, messages and support tickets in many languages. It is Apache-2.0 and trained only on openly licensed text with labels from an Apache-2.0 LLM, so it can be used commercially. It includes ONNX files for CPU and the browser (transformers.js). A smaller, faster version is available as multilingual-sentiment-small. Try it in the browser.

• Mixed or balanced opinions ("good quality but too expensive") are neutral. Plain facts, questions and requests are neutral too.
• The scores are probabilities, e.g. positive - negative gives a -1 to 1 polarity score.
• onnx/model_quantized.onnx (int8 embeddings, 641 MB) gives the same label as fp32 on 99.5% of 430 benchmark texts.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Horizon-Labs/multilingual-sentiment-base")
clf(["I love this phone, the camera is amazing!", "Le colis est arrivé cassé.", "Der Termin ist am Montag."])
# [{'label': 'positive', ...}, {'label': 'negative', ...}, {'label': 'neutral', ...}]

transformers.js:

import { pipeline } from "@huggingface/transformers";
const clf = await pipeline("text-classification", "Horizon-Labs/multilingual-sentiment-base", { dtype: "q8" });
console.log(await clf("Obrigado pela ajuda, vocês são incríveis!"));

Evaluation

These are public benchmarks, used only for evaluation. We never trained on them, and any training text that also appears in them was removed. The metric is macro-F1, averaged over the languages of each benchmark. On the two-class MTEB sets, each model's negative-vs-positive choice is scored, ignoring neutral. Every model is run with the same script on the same texts (code/), and each model's labels are mapped to negative/neutral/positive (1-2 stars = negative, 3 = neutral, 4-5 = positive; "very negative" = negative).

• Numbers are for the released checkpoint (seed 0). Mean of two training seeds: small tweets .633 / Amazon .690 / MTEB .824; base tweets .658 / Amazon .703 / MTEB .855.
• Tweets: cardiffnlp/tweet_sentiment_multilingual test (870 per language). Amazon: amazon_reviews_multi test (900 per language, balanced; 3 stars = neutral). MTEB: mteb/multilingual-sentiment-classification test (up to 600 per language; several languages are machine-translated, e.g. Welsh = translated IMDB).
• Models ahead of this one: tweets: twitter-xlm-roberta-base-sentiment, twitter-xlm-roberta-base-sentiment-multilingual, modernBERT-base-multilingual-sentiment; Amazon: modernBERT-base-multilingual-sentiment; MTEB: none. Models trained on a benchmark's own training split have an in-domain advantage on it; this model has seen no benchmark data.

Per benchmark language:

Training

• Text (about 433k training examples): snippets from FineWeb-2 and FineWeb (ODC-BY), in 67 languages, half of them from review, forum, comment and blog pages. We added synthetic reviews, posts, comments, messages and complaints in about 90 languages and varieties (including Hinglish and Arabic dialects), written by Qwen3.8-27B (Apache-2.0) in many genres, topics and lengths.
• Labels: soft labels (probabilities for negative, neutral and positive) from Qwen3.8-27B with a fixed zero-shot instruction (code/sentiment/teacher_sent.py). The model is trained to match these probabilities. The teacher's scores on the benchmarks are in the table above.
• Model: mmBERT-base with a 3-way head, 3 epochs, max length 512 tokens. The checkpoint was chosen by agreement with the teacher on held-out teacher-labelled data, never on the benchmarks.
• Code: code/ in this repository.

Limitations

• Labels come from an LLM, not from human annotators, so the model inherits the teacher's judgement. Its idea of "neutral" can differ from a given dataset's convention: tweet benchmarks mark many mildly opinionated posts as neutral.
• Accuracy is lower on sarcasm, on very short or context-dependent messages (tweets), and on some low-resource languages. Benchmark sets below 0.70 macro-F1: amazon_zh, mteb_bam, mteb_uig, tweets_ar, tweets_fr, tweets_ge, tweets_hi, tweets_po, tweets_sp.
• Trained on texts up to 512 tokens and evaluated on texts up to 2,000 characters. The model accepts up to 8,192 tokens, but longer documents are untested; for those, pass truncation=True, max_length=512, or score paragraphs separately.
• Sentiment is not stance, emotion or toxicity. Don't use it to make decisions about individuals.