Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs: a systematic evaluation

Software languages evolve over time for various reasons, such as the addition of new features. When the language’s grammar definition evolves, textual instances that originally conformed to the grammar become outdated. For DSLs in a model-driven engineering context, there exists a plethora of techniques to co-evolve models with the evolving metamodel. However, these techniques are not geared to support DSLs with a textual grammar—applying them to textual language definitions and instances may lead to the loss of information from the original instances, such as layout information and comments, which are valuable for software comprehension and maintenance. This study systematically evaluates the potential of Large Language Model (LLM)-based solutions in achieving grammar and instance co-evolution for textual DSLs. By applying two advanced language models, Claude Sonnet 4.5 and GPT-5.2, and conducting ten experimental runs per case across ten case languages, we evaluate both the correctness of co-evolved instances and the preservation of human-oriented information such as comments and layout. Our results indicate high performance on small-scale cases ($\geq$94% precision and recall for instances with fewer than 20 lines requiring modification), but performance degraded with scale: Claude Sonnet 4.5 maintained 85% recall at 40 lines while GPT-5.2 showed greater sensitivity, failing entirely on the two largest instances. ... mehrInstance scale also substantially impacts processing efficiency, with Claude’s response time increasing nearly 18-fold for the largest case. In addition, we observe that grammar evolution complexity and deletion granularity impact performance more than change type alone, and prompt transferability across LLMs is limited. These findings identify the conditions under which LLM-based co-evolution succeeds and its scalability limitations, providing practitioners with insights into applicability and researchers with directions for addressing current limitations.