KIT | KIT-Bibliothek | Impressum | Datenschutz

Exploring the Robustness of the Natural Language Inference Capabilties of T5

Grötzinger, Dennis

Abstract (englisch):

Large language models like T5 perform excellently on various NLI benchmarks. However, it has been shown that even small changes in the structure of these tasks can significantly reduce accuracy. I build upon this insight and explore how robust the NLI skills of T5 are in three scenarios. First, I show that T5 is robust to some variations in the MNLI pattern, while others degenerate performance significantly. Second, I observe that some other patterns that T5 was trained on can be substituted for the MNLI pattern and still achieve good results. Third, I demonstrate that the MNLI pattern translate well to other NLI datasets, even improving accuracy by 13% in the case of RTE. All things considered, I conclude that the robustness of the NLI skills of T5 really depend on which alterations are applied.

Volltext §
DOI: 10.5445/IR/1000139678
Veröffentlicht am 10.11.2021
Cover der Publikation
Zugehörige Institution(en) am KIT Institut für Informationssicherheit und Verlässlichkeit (KASTEL)
Publikationstyp Hochschulschrift
Publikationsdatum 21.06.2021
Sprache Englisch
Identifikator KITopen-ID: 1000139678
Art der Arbeit Abschlussarbeit - Bachelor
Prüfungsdaten 21.06.2021
Referent/Betreuer Keim, J.
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page