Code and publicly shareable analysis materials for the manuscript by Dhruv Agarwal, Anya Shukla, Sunayana Sitaram, and Aditya Vashistha.
We evaluate seven Indic and five global reference LLMs with population-scale surveys, community-sourced benchmarks, and a writing experiment with 115 Indian participants. Across the evaluations, language-focused regionalization does not consistently improve alignment with Indian values or practices.
Negative normalized Cultural Alignment Differential (nCAD) indicates greater similarity to US than Indian public-opinion distributions. Error bars are 95% confidence intervals.
| Dimension | Evaluation | Main code |
|---|---|---|
| Values | Inglehart-Welzel cultural map | cultural_map/ |
| Values | GlobalOpinionQA opinion alignment | opinions/ |
| Practices | Sanskriti cultural knowledge | sanskriti/ |
| Practices | CulturalBench cultural knowledge | culturalbench/ |
| Practices | DIWALI cultural adaptation | diwali/ |
| Practices | NormAd cultural adaptation | normad/ |
| Downstream study | AI-assisted writing | writing_study/ |
The repository contains prompts, model identifiers, inference code, analysis notebooks, and statistical tests. Large model outputs and third-party datasets are not duplicated; the code downloads or identifies the original datasets. Raw essays and interaction logs are not released because participants did not consent to public dissemination of their free-text responses.
The full benchmark runs require Linux, NVIDIA GPUs, access to gated model checkpoints, and substantial compute. Python 3.11 is the tested environment.
git clone https://github.com/agdhruv/fluent-foreign.git
cd fluent-foreign
uv sync --frozen
cp .env.example .env
Set HF_TOKEN only for gated Hugging Face models and OPENAI_API_KEY only for scripts that call OpenAI. Do not commit .env or generated outputs.
See docs/REPRODUCIBILITY.md for the experiment-by-experiment execution map, docs/MODELS.md for exact model identifiers, and docs/DATA.md for data provenance and the writing-study privacy boundary.
Depending on the task, models were evaluated with:
Every benchmark script records or constructs its prompt in the corresponding directory. The writing-study autocomplete prompts are centralized in writing_study/prompts.py.
The manuscript does not yet have a final DOI. Until then, use the metadata in CITATION.cff and cite this repository URL. The citation metadata should be updated when the archival paper record is available.
Benchmark data comes from the original public releases cited in the manuscript. The writing experiment was approved by Cornell University's Institutional Review Board (protocol IRB0148401). Only aggregate results, prompts, schemas, and analysis code are public here; participant-authored text and complete logs are excluded.
Code and publicly shareable analysis materials for the manuscript by Dhruv Agarwal, Anya Shukla, Sunayana Sitaram, and Aditya Vashistha.
We evaluate seven Indic and five global reference LLMs with population-scale surveys, community-sourced benchmarks, and a writing experiment with 115 Indian participants. Across the evaluations, language-focused regionalization does not consistently improve alignment with Indian values or practices.
Negative normalized Cultural Alignment Differential (nCAD) indicates greater similarity to US than Indian public-opinion distributions. Error bars are 95% confidence intervals.
| Dimension | Evaluation | Main code |
|---|---|---|
| Values | Inglehart-Welzel cultural map | cultural_map/ |
| Values | GlobalOpinionQA opinion alignment | opinions/ |
| Practices | Sanskriti cultural knowledge | sanskriti/ |
| Practices | CulturalBench cultural knowledge | culturalbench/ |
| Practices | DIWALI cultural adaptation | diwali/ |
| Practices | NormAd cultural adaptation | normad/ |
| Downstream study | AI-assisted writing | writing_study/ |
The repository contains prompts, model identifiers, inference code, analysis notebooks, and statistical tests. Large model outputs and third-party datasets are not duplicated; the code downloads or identifies the original datasets. Raw essays and interaction logs are not released because participants did not consent to public dissemination of their free-text responses.
The full benchmark runs require Linux, NVIDIA GPUs, access to gated model checkpoints, and substantial compute. Python 3.11 is the tested environment.
git clone https://github.com/agdhruv/fluent-foreign.git
cd fluent-foreign
uv sync --frozen
cp .env.example .env
Set HF_TOKEN only for gated Hugging Face models and OPENAI_API_KEY only for scripts that call OpenAI. Do not commit .env or generated outputs.
See docs/REPRODUCIBILITY.md for the experiment-by-experiment execution map, docs/MODELS.md for exact model identifiers, and docs/DATA.md for data provenance and the writing-study privacy boundary.
Depending on the task, models were evaluated with:
Every benchmark script records or constructs its prompt in the corresponding directory. The writing-study autocomplete prompts are centralized in writing_study/prompts.py.
The manuscript does not yet have a final DOI. Until then, use the metadata in CITATION.cff and cite this repository URL. The citation metadata should be updated when the archival paper record is available.
Benchmark data comes from the original public releases cited in the manuscript. The writing experiment was approved by Cornell University's Institutional Review Board (protocol IRB0148401). Only aggregate results, prompts, schemas, and analysis code are public here; participant-authored text and complete logs are excluded.