Incremental learning of humanoid robot behavior from natural interaction and large language models

Natural-language dialog is key for an intuitive human–robot interaction. It
can be used not only to express humans’ intents but also to communicate
instructions for improvement if a robot does not understand a command
correctly. It is of great importance to let robots learn from such interaction
experiences in an incremental way to allow them to improve their behaviors
or avoid mistakes in the future. In this paper, we propose a system to
achieve such incremental learning of complex high-level behavior from natural
interaction and demonstrate its implementation on a humanoid robot. Our
system deploys large language models (LLMs) for high-level orchestration of
the robot’s behavior based on the idea of enabling the LLM to generate Python
statements in an interactive console to invoke both robot perception and action.
Human instructions, environment observations, and execution results are fed
back to the LLM, thus informing the generation of the next statement. Since an
LLM can misunderstand (potentially ambiguous) user instructions, we introduce
incremental learning from the interaction, which enables the system to learn
... mehr
from its mistakes. For that purpose, the LLM can call another LLM responsible for
code-level improvements in the current interaction based on human feedback.
Subsequently, we store the improved interaction in the robot’s memory so that it
can later be retrieved on semantically similar requests. We integrate the system in
the robot cognitive architecture of the humanoid robot ARMAR-6 and evaluate
our methods both quantitatively (in simulation) and qualitatively (in simulation
and real-world) by demonstrating generalized incrementally learned knowledge