Local LLMs for End-To-End Testing in Practice: Lessons from a Smart City Web Application

End-to-End (E2E) testing of modern web applications simulates real user interactions to verify the complete application flow, ensuring seamless integration of UI, functionality, and data. It is, in general, costly to create and maintain, especially when the backend is a smart city platform with a data lake and complex user flows that introduce additional complexity. Recent advances in Large Language Models (LLMs) offer new opportunities for automating parts of this process, but their practical adoption in real development environments remains challenging, particularly when privacy, monetary cost, deployment, and control requirements call for local rather than cloud-based models.
This paper investigates the feasibility of introducing local LLMs into the E2E testing process of a real-world smart city web application, the ENEA PELL-IP portal for monitoring public lighting infrastructures. We developed and evaluated GenE2E, a two-stage pipeline that generates test cases from use case specifications and transforms them into executable Playwright tests. The approach was integrated with an existing testing environment and evaluated against manually written baseline tests.
... mehr
Our findings show that local LLMs can effectively support E2E test generation, but not yet as fully autonomous tools. In our setting, Llama 3.3 outperformed CodeLlama mainly due to its larger context window; single-test generation improved focus and coverage, whereas batch processing reduced model invocations and showed lower generation time in our setup, but the timing results are hardware-dependent and based on a single run; broader efficiency claims require repeated measurements on representative hardware. We also observed that specification quality, completeness of Page Object Model (application-specific API wrapping HTML pages), and use case dependency management had a major impact on generated test quality. Based on this feasibility study, we derive preliminary lessons for practitioners considering local LLMs for E2E test generation.