Generalizable Representations for Cross-Lingual Transfer Learning

Multilingual models are essential in natural language processing (NLP),
the field of enabling computational models to process and generate human language.
These models are crucial for expanding information access across linguistic boundaries,
with machine translation models being particularly central to this effort.
However, with approximately 7,000 languages spoken worldwide,
it is infeasible to acquire sufficient training data for every language.
This data scarcity challenge highlights the importance of cross-lingual generalization,
the ability to transfer knowledge learned from some languages to settings with little or no training data.
Such generalization depends on a model's ability to learn internal representations that are shared and consistent across languages.

While the \textit{classical} form of cross-lingual generalization involves training a model for a task in some languages and evaluating it on others,
its application in machine translation is more complex as the input and output languages are inherently different.
The directional nature of machine translation calls for \textit{compositional} generalization,
... mehr
which requires the model to combine knowledge from trained directions (e.g., English-to-German and German-to-French) to directly perform translation for unseen directions (e.g., English-to-French).
Moreover,
the pre-training fine-tuning paradigm introduces a crucial \textit{temporal} dimension to generalization.
This perspective concerns how well a model's pre-trained capabilities are preserved after specializing on selected tasks or languages.
This thesis addresses these three aspects of generalization: classical, compositional, and temporal.

First, we investigate compositional generalization in translation models trained from scratch.
We demonstrate that a primary reason why multilingual translation models fail to generalize compositionally is that their internal representations become highly entangled with information specific to the input language.
We identify that this entanglement is reinforced by the residual connections within the model's architecture.
By proposing a method that relaxes this structural constraint, we achieve substantial improvements on unseen translation directions.

We then extend the investigation to the temporal dimension of generalization by incorporating pre-trained models.
We observe that despite large-scale multilingual pre-training,
a model's internal representations often remain highly specific to the input language.
Moreover,
when these models are specialized for particular translation directions,
they are prone to catastrophic forgetting, which substantially degrades their existing capabilities.
To counteract this, our work evaluates several established mitigation strategies from the literature and introduces a novel approach.
Our proposed method produces a more interpretable view of the model's internal representations,
although this benefit comes with a trade-off in translation quality.

Our investigation then extends to large language models (LLMs), which operate as general-purpose instruction-following systems.
For these models, which typically use a decoder-only architecture,
we first identify the middle layers as the most effective positions for intervention.
Building on this finding, we propose a modular and computationally efficient post-hoc method to enhance language invariance in the model's representations.
This approach leads to substantial improvements in both classical cross-lingual transfer scenarios and the compositional generalization cases examined earlier.

Lastly, we revisit classical cross-lingual transfer with a novel application for attribute-controlled machine translation.
This method trains a classifier to distinguish text attributes in languages with available labeled data and then transfers this capability to guide translations for other languages lacking such resources.
The approach operates on the model's internal activations rather than weights,
making it suitable for scenarios where model weights must remain fully frozen.
When compared to an oracle setup that updates the model weights,
our activation-based method demonstrates stronger robustness against domain shifts.

Each of the contributions above is situated within a model modification scope that aligns with evolving practices in the field,
ranging from training models from scratch to performing post-hoc interventions on large pre-trained models.
This thesis advances the development of multilingual NLP models that not only achieve strong performance under trained conditions but also acquire more generalizable and adaptable underlying representations.