Taming Large Language Models via Scripted Interactions

We are currently experiencing a bloom of Artificial Intelligence (AI)-powered software thanks to the development and advancements of
Large Language Models (LLMs). Despite the applications of these LLMs being impressive and countless, these tools alone are unreliable. In fact, the possibility of LLMs generating faulty or hallucinated content makes them unsuitable for automating workflows and pipelines. In this context, Software Engineering (SE) comes in handy, offering a wide variety of formal tools to specify, verify and validate software behaviour. These SE tool can be used to describe constraints over an LLM output and, thus, provide better guarantees on the generated content. In this paper, we argue how the development of a Domain Specific Language (DSL) for scripting interactions LLMs, namely LLM Scripting Language (LSL), could be the key to improve AI-based applications. The idea in the long term is to make the interaction with LLMs programmable, abstracting from their implementation and training procedure. Nowadays, LLMs and LLM-based software are still lacking in terms of reliability, robustness, and trustworthiness. With LSL, we aim at tackling these deficiencies by looking into approaches to control the output of LLMs, to impose a structure to the interaction and to combine these ideas with verification, validation, and explainability.