KIT | KIT-Bibliothek | Impressum | Datenschutz

GraSAME: Injecting Token-Level Structural Information to Pretrained Language Models via Graph-guided Self-Attention Mechanism

Yuan, Shuzhou 1; Färber, Michael ORCID iD icon 1
1 Institut für Angewandte Informatik und Formale Beschreibungsverfahren (AIFB), Karlsruher Institut für Technologie (KIT)

Abstract:

Pretrained Language Models (PLMs) benefit from external knowledge stored in graph structures for various downstream tasks. However, bridging the modality gap between graph structures and text remains a significant challenge. Traditional methods like linearizing graphs for PLMs lose vital graph connectivity, whereas Graph Neural Networks (GNNs) require cumbersome processes for integration into PLMs. In this work, we propose a novel graph-guided self-attention mechanism, GraSAME. GraSAME seamlessly incorporates token-level structural information into PLMs without necessitating additional alignment or concatenation efforts. As an end-to-end, lightweight multimodal module, GraSAME follows a multi-task learning strategy and effectively bridges the gap between graph and textual modalities, facilitating dynamic interactions between GNNs and PLMs. Our experiments on the graph-to-text generation task demonstrate that GraSAME outperforms baseline models and achieves results comparable to state-of-the-art (SOTA) models on WebNLG datasets. Furthermore, compared to SOTA models, GraSAME eliminates the need for extra pre-training tasks to adjust graph inputs and reduces the number of trainable parameters by over 100 million.

Zugehörige Institution(en) am KIT Institut für Angewandte Informatik und Formale Beschreibungsverfahren (AIFB)
Publikationstyp Proceedingsbeitrag
Publikationsjahr 2024
Sprache Englisch
Identifikator ISBN: 979-88-917611-9-3
KITopen-ID: 1000172732
Erschienen in Findings of the Association for Computational Linguistics: NAACL 2024
Veranstaltung Findings of the Association for Computational Linguistics (2024), Mexiko-Stadt, Mexiko, 16.06.2024 – 21.06.2024
Verlag Association for Computational Linguistics (ACL)
Seiten 920 – 933
Nachgewiesen in OpenAlex
Dimensions
Scopus
Relationen in KITopen

Verlagsausgabe §
DOI: 10.5445/IR/1000172732
Veröffentlicht am 30.07.2024
Originalveröffentlichung
DOI: 10.18653/v1/2024.findings-naacl.58
Seitenaufrufe: 32
seit 30.07.2024
Downloads: 22
seit 30.07.2024
Cover der Publikation
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page