🐪 Dataset Card for Dromedario 3

Dataset Description

• Paper: TBA
• Point of Contact: {first-author-surname} {at} diag.uniroma1.it

📋 Dataset Summary

Dromedario 3 is a large-scale Italian instruction-tuning dataset derived from the English Tülu 3 SFT mixture through a principled translation pipeline: instructions and responses are classified according to the Natural Instructions taxonomy, classes are manually reviewed for translation safety, and items in validated classes are machine-translated into Italian. The full procedure is described in Initial Data Collection and Normalization.

Dromedario 3 is intended for supervised fine-tuning of Italian (and Italian/English bilingual) large language models, supporting the same broad range of tasks as Tülu 3 — open-ended dialogue, reasoning, coding, and knowledge-intensive instruction following — with Italian instruction-response pairs alongside the original English data.

Every row carries both the original conversation and, where available, its Italian translation, so English-only, Italian-only, and mixed training sets can all be built from this single release:

from datasets import load_dataset

ds = load_dataset("sapienzanlp/dromedario-3-sft-dataset", split="train")

# Italian subset: rows with a usable translation
it_only = ds.filter(lambda x: x["messages_transl"] is not None)

# Optionally drop rows that were automatically corrected
it_safe = it_only.filter(lambda x: not x["flags"])

🏆 Supported Tasks and Leaderboards

• text-generation: The dataset can be used to fine-tune a model for instruction following, which consists in generating a helpful, contextually appropriate response given a natural language instruction (single-turn or multi-turn). Models fine-tuned on Dromedario 3 are evaluated on ITA-Bench, a suite of nine benchmarks covering both multiple-choice and open-ended tasks in Italian and English. In our experiments, Llama 3.1 8B and Minerva 7B were fine-tuned on mixtures of Dromedario 3 and the original Tülu 3 SFT data.

🌐 Languages

Dromedario 3 contains primarily Italian (it) and English (en) instances, reflecting its origin as a partial translation of the Tülu 3 SFT mixture. Some source datasets contain inherently multilingual data (e.g., aya), so other languages may also be present in smaller amounts.

Untranslated items retain instructions and responses in their original language, while translated items contain Italian translations of the original English instructions and responses. We do not target languages other than English for translation, but some items erroneously labeled as en may have been translated as a result.

Code-switching, code blocks, math, and translation requests occur throughout the dataset. The text is primarily standard written Italian and English, spanning the range of registers and domains present in Tülu 3 (e.g. conversational, technical, code-related, and formal instructional text), rather than a specific regional or social dialect.

🧩 Dataset Structure

Data Instances

A typical instance looks like:

{
  "id": "oasst1_888",
  "source": "ai2-adapt-dev/oasst1_converted",
  "task": "Question Answering",
  "domain": "Natural Science",
  "lang_orig": "en",
  "messages_orig": [
    {"role": "user", "content": "What's the weather like?"},
    {"role": "assistant", "content": "Could you please specify a location or area you would like me to describe the weather for?"}
  ],
  "messages_transl": [
    {"role": "user", "content": "Com'è il tempo?"},
    {"role": "assistant", "content": "Potresti specificare la località o l'area di cui vuoi sapere il tempo?"}
  ],
  "flags": []
}

This is a two-turn conversation (user/assistant), the common case: messages_orig and messages_transl are parallel arrays of the same length, same role sequence, turn-for-turn, while flags is empty here since no issues were found in this example.

Data Fields

• id (string): unique identifier for the conversation, shared with allenai/tulu-3-sft-mixture.
• source (string): name of the original sub-dataset this row came from (e.g., ai2-adapt-dev/numinamath_tir_math_decontaminated, allenai/tulu-3-sft-personas-math-grade).
• task (string): the instruction's task category (e.g. "Question Answering", "Mathematics", "Coding"). Synthetic label based on the Natural Instructions taxonomy, generated by GPT-oss-120b.
• domain (string): the instruction's subject-matter category (e.g. "Natural Science", "Mathematics"). Synthetic label based on the Natural Instructions taxonomy, generated by GPT-oss-120b.
• lang_orig (string): the language code of messages_orig's content. Automatic label, not always accurate.
• messages_orig (list[{"role": string, "content": string}]): the original conversation, turn by turn. Always present.
• messages_transl (list[{"role": string, "content": string}] | null): the Italian translation of messages_orig, same turn structure. null means no usable Italian translation exists, for either of two reasons, distinguishable only via flags: the row was never translated (flags is empty), or it was translated and then discarded as unrecoverably broken (flags contains a *_discarded tag).
• flags (list[string]): quality-control tags from the translation pipeline, listed below. Empty for rows that passed all checks unmodified and for rows that were never translated.

Flag vocabulary

Note on the flags: these flags are added during the last refinement stage, after the postprocessing described in the paper (see Post-translation quality control).

Rows touched by nothing have an empty flags list, as do rows that were never translated. 13,080 rows (2.08% of the translated portion) carry at least one flag.

Note on id and source

• Every id in this dataset is the exact same id used in allenai/tulu-3-sft-mixture, and messages_orig is byte-identical to that source's own messages field for the same id.
• id does not reliably encode source, as in the original dataset. Some ids carry a literal source-name prefix (ai2-adapt-dev/tulu_v3.9_aya_100k_46337), others don't (personahub_ss6dri12ftk0e4v0t59vbgyy → source = "ai2-adapt-dev/personahub_math_v5_regen_149960"). Always group or filter by source, never by parsing id.

Data Splits

This dataset has a single split, train, with 939,103 rows.

The translated count is net of quality control: 627,817 rows were translated and scored, and 5,137 were discarded as unrecoverable (see Post-translation quality control). Discarded rows are not removed from the dataset — they are retained with messages_transl = null and a *_discarded flag, so the row count above is unchanged.

🛠️ Dataset Creation

Curation Rationale

Italian-native instruction-tuning data remains scarce relative to English, and existing Italian resources are typically undocumented, either small standalone releases or byproducts of model-specific training mixtures, making systematic comparison and reuse difficult. Dromedario 3 addresses this gap by translating a principled subset of the English Tülu 3 SFT mixture, prioritizing provenance and traceability over raw scale: rather than maximizing translated volume, the dataset applies taxonomy-based filtering to select instructions and responses that remain meaningful after translation into Italian, discarding classes where translation would be unsafe or would alter task semantics (e.g. English-specific linguistic tasks). This selective approach is deliberately incomplete: partial coverage of Tülu 3 enables controlled experiments on mixing ratios between untranslated English and translated Italian data during fine-tuning, rather than treating full translation coverage as a goal in itself.

Source Data

Dromedario 3 is derived entirely from the Tülu 3 SFT mixture, an English instruction-tuning dataset combining human-written, synthetic, and model-generated instructions and responses from a variety of source datasets (e.g. aya). Each instance is inherited from Tülu 3 as-is. No new instructions or responses were authored specifically for Dromedario 3; all content originates from Tülu 3's existing sources.

Initial Data Collection and Normalization

Dromedario 3 starts from the Tülu 3 SFT mixture in its entirety. Each instance (instruction and response) is first classified according to the Natural Instructions taxonomy using GPT-oss-120b. Classes are then manually reviewed to determine whether their instances are safe to translate into Italian without altering task semantics (e.g. classes built around English-specific linguistic phenomena are excluded). Instances belonging to validated classes are translated into Italian using TranslateGemma-27B.

Because TranslateGemma-27B does not support system prompts and tends to execute the instruction it is given rather than translate it, instances are reformatted before translation using a dialogue-style prefix combined with generation prefilling, which reliably steers the model into translation behavior rather than task completion. See our paper for more information on how we prompted the model.

Note on the translation model. We used the Infomaniak build of TranslateGemma-27B, as it was supported by vLLM.

Post-translation quality control

After the postprocessing described in the paper, which accounts for the majority of the problems, we proceeded to refine the dataset at a deeper level. We focused on errors such as: generation stops mid-response, a section of content is silently dropped, or an English keyword inside a code block is replaced by its Italian gloss, leaving code that no longer parses.

Every translated row is passed through automatic detectors targeting these failure modes, after which it is discarded, corrected, or tagged for downstream filtering.

Failure rates are concentrated: truncation is almost entirely confined to personahub_math_v5_regen_149960 (1.72%, against 0.00% for most sources), and Python code corruption runs at 1.79% of blocks against 0.56% across all other languages combined.

The confidence of this last quality control is not uniform; truncation, omission, and \boxed{} counts are exact, as the Python code-corruption rate, since every block is checked with ast.parse. The non-Python rate rests on a per-language keyword lookup rather than a parser, and should be read as a lower bound.

Note: these checks postdate our paper. Results reported there were obtained before this stage was applied and, even though the amount of affected items is low (~2%), there might be discrepancies.

Per-source discard / retention breakdown

• Translated — rows with a translation attempt (retained + discarded).
• Truncation discarded / Omission discarded — translation dropped as unrecoverable; the row still ships with messages_transl = null and the matching flag.
• Total discarded — union of the two.
• Retained — rows shipping with a usable Italian translation.
• Retained-then-corrected — retained rows where at least one correction was applied (boxed_corrected, code_keyword_fixed, nonpython_code_swapped, omission_corrected). Excludes no_self_correction and code_keyword_unresolved, which flag a row without changing its text.

Who are the source language producers?

The original instructions and responses are inherited from the Tülu 3 SFT mixture, which combines data from multiple source datasets produced by a mix of human annotators/crowdworkers and machine-generated content (e.g. synthetic instructions and responses generated by large language models, subsequently filtered or curated). We refer readers to the Tülu 3 SFT mixture dataset card and its cited source datasets for the exact proportion of human- vs. machine-produced content, as well as any self-reported demographic information about human annotators.

The Italian portion of Dromedario 3 is additionally produced by machine translation, with no human translators involved in generating the Italian text itself. Manual human review is limited to class-level validation (determining whether a class is safe to translate) and instance-level translation quality checks, rather than authoring or translating individual instances.

Annotations

Dromedario 3 adds taxonomy-based class labels to each instance, used to determine translation eligibility, which are not part of the original Tülu 3 SFT mixture.

Annotation process

Each instance (instruction and response pair) is classified according to the Natural Instructions taxonomy using GPT-oss-120b, served via vLLM. The resulting class labels are then manually reviewed by two annotators to determine whether instances in that class are safe to translate into Italian without altering task semantics; this review is conducted at the class level and by inspecting a subsample of individual instances for each class. Annotators judge GPT-oss-120b's labels as correct in the large majority of cases, with domain labels more reliable than task labels (~95% vs ~87%). Inter-annotator agreement is substantial for both dimensions, with Cohen's κ of 0.80 for task-type labels and 0.75 for domain labels.

Who are the annotators?

Class labels are machine-generated by GPT-oss-120b. The subsequent validation step, i.e., determining which classes are safe to translate, is performed by two of the paper's authors, who are NLP researchers and linguists affiliated with Sapienza University of Rome with native or near-native proficiency in both Italian and English. No external crowdworkers or third-party annotators were involved, and no compensation was applicable.

Personal and Sensitive Information

We refer readers to the Tülu 3 SFT mixture dataset card for details on personal and sensitive information handling in the source data. No additional anonymization process was applied during translation or filtering.

⚠️ Considerations for Using the Data

Social Impact of Dataset

Dromedario 3 aims to reduce the gap in instruction-tuning resources between English and Italian. While Italian is not an extremely low-resource language overall (it has substantial monolingual and parallel corpora), it remains comparatively underrepresented in the instruction-tuning and alignment data used to build modern LLMs, most of which is produced natively in English. By providing a large, documented, traceable, principled translation of an established English SFT mixture, Dromedario 3 supports the development of Italian and Italian/English bilingual models without requiring every research group to build their own translation pipeline from scratch, lowering the barrier to entry for Italian-focused NLP research.

Positive impacts include improved accessibility of instruction-following LLMs for Italian speakers, particularly for use cases (education, public services, assistive technologies) where native-language interaction matters and where reliance on English-centric models can degrade quality or naturalness of responses. Making the dataset and its construction pipeline public also supports reproducibility and comparison across Italian LLM efforts, addressing a fragmentation described in the Curation Rationale, where existing resources are often small, standalone, or tied to a single model's training run.

Risks include the propagation of biases and errors already present in Tülu 3 into the Italian-language ecosystem, potentially at larger scale, since translation does not filter out biases in content, only in translatability. Machine translation may also introduce systematic errors, unnatural phrasing, or subtle shifts in meaning that are harder to detect for Italian users than for the English-fluent researchers who validated the source data. Additionally, as with any instruction-tuning dataset, models trained on Dromedario 3 could be used to automate tasks or generate content in ways that affect employment, misinformation, or decision-making processes, and the translation process itself does not mitigate these risks beyond what already applies to Tülu 3.

Discussion of Biases

Dromedario 3 inherits any biases already present in the Tülu 3 SFT mixture (e.g. topical, cultural, or demographic skews originating from its source datasets), since translation does not filter for or correct such biases; classification and translation criteria are based on task and domain safety, not on identifying or mitigating social bias. No dedicated bias auditing has been performed on Dromedario 3 specifically.

Translation introduces additional, translation-specific biases. Content translated from English by TranslateGemma-27B may reflect a "translationese" register, with sentence structures, idioms, or phrasing that are grammatically correct Italian but not how Italian speakers would naturally phrase the same content, unlike Italian instruction data collected natively. We carried out a preliminary comparison of candidate translation models, and to the best of our knowledge, TranslateGemma-27B appears to be the best model for this task, but no quantitative study of the resulting translationese bias itself has been conducted.

Other Known Limitations

• Partial coverage. Dromedario 3 does not translate the full Tülu 3 mixture; classes deemed unsafe or semantically altered by translation are excluded. This is a deliberate design choice (see Curation Rationale) but means the dataset does not offer a complete Italian counterpart to Tülu 3.
• Translation quality is not exhaustively validated. Manual review targeted class-level translation safety, not per-instance translation quality. Automatic checks catch structural failures (truncation, omitted content, corrupted code, lost \boxed{} formatting) but not semantic ones: a fluent, complete, well-formed mistranslation passes every detector.
• Code-corruption detection has known recall limits outside Python. Non-Python blocks are screened with a per-language keyword lookup rather than a parser, so the 0.56% corruption rate for those languages is a lower bound. Rows relying on non-Python code should not be assumed clean because they carry no flag.

ℹ️ Additional Information

Dataset Curators

Dromedario 3 is the result of a group effort and many brainstorming sessions: we detail here our main contributions.

• Luca Gioffré: project lead, data curator, SFT training
• Marina Iuliana Aur: project lead, data curator
• Francesco Ortame: data classification
• Luca Moroni: benchmarking
• Alberte Fernández-Castro: annotator
• Elena Marafatto: annotator
• Roberto Navigli: project supervisor

Licensing Information

Dromedario 3 is released under the ODC-BY 1.0 license, matching the license of the source Tülu 3 SFT mixture. As with Tülu 3, this is a mixture license: different licenses apply to the underlying source datasets that Tülu 3 (and therefore Dromedario 3) draws from, and some portions of the data are non-commercial. Dromedario 3 is presented as a research artifact, consistent with the licensing terms of its source data.

The Italian portion of Dromedario 3 was generated with TranslateGemma-27B, distributed under the Gemma Terms of Use. Under those terms, model Outputs are explicitly not considered Model Derivatives, and Google claims no rights in Outputs generated using Gemma (Section 3.3).

Citation Information

TBA

@inproceedings{gioffre-2026-dromedario3,
  title     = {Dromedario-3: Localizing T{\"u}lu 3 Dataset to Italian},
  author    = {Gioffr{\`e}, Luca and Aur, Marina Iuliana and Ortame, Francesco and Moroni, Luca and Fern{\'a}ndez-Castro, Alberte and Marafatto, Elena and Navigli, Roberto},
  booktitle = {Proceedings of the Twelfth Italian Conference on Computational Linguistics (CLiC-it 2026)},
  editor    = {Basile, Valerio and Croce, Danilo and Passaro, Lucia C. and Pirrone, Roberto},
  year      = {2026},
  address   = {Palermo, Italy},
  month     = sep
}