Language Similarity

This is a project focused on evaluating and interpreting how LLMs concepualize language and implement translation.

At the start, we was focused on quantifying the similarity of languages using a few different metrics:

• Representation Similarity (cosine similarity of the average activation vector across different languages)
• Typological Similarity (Jaccard similarity of feature vectors derived from WALS syntax datasets)
• Translation Similarity (BLEU score of translation on the FLORES dataset. Proxy for how 'easy' it is for the model to move between two languages)

Currently, we're working on comparing the two hypotheses for LLM translation:

• Noisy Channel: when translating from French to English, French text is encoded into a latent space in the middle layers of the model, then decoded to English. Nearly all features learned by the model in the latent space are universal between languages sharing those grammatical concepts.
• Translation Module: although the model uses a noisy channel for many operations, pairwise abstractions are learned alongside the noisy channel. For instance, the existence of a formality marker in Polish may be directly mapped to the existence of a formality marker with similar connotations in Japanese.