Benchmarking Agentic AI Architectures for Automation Engineering

Developing automation software requires not only control-logic generation but also hardware configuration, I/O mapping, and interaction with vendor-specific engineering tools.
Despite the success of large language models (LLMs) and agentic AI in conventional software engineering, their utility for end-to-end automation engineering is under-explored.
This paper compares seven architectures, ranging from a manual LLM baseline to standardized tool-enabled single-agent and multi-agent systems with optional retrieval-augmented generation and inter-agent communication.
The benchmark is evaluated on two industrial use cases and measures syntactic correctness and semantic correctness against predefined hardware test cases, tool calls, token consumption, and cost.
The results show that explicit orchestration and in particular access to the vendor-specific engineering tool substantially improves functional performance over plain LLM baselines.
Single-agent setups using vendor-specific tools and closed-loop orchestration performed best. Multi-agent and RAG enhancements provided negligible gains, likely because the current task complexity does not justify their additional coordination overhead.
... mehr
These findings indicate that lightweight agent designs are currently a practical choice for industrial automation engineering, while standardized tool and inter-agent protocols remain promising for modularity and extensibility.