Interpretable World Model Imaginations as Deep Reinforcement Learning Explanation

Explainable Deep Reinforcement Learning aims to clarify the decision-making processes of agents. Recent world model-based approaches, such as Dreamer, train agents through “imagination,” where the actor learns by interacting with a learned world model that simulates the environment. Consequently, the overall performance of these systems depends not only on the learned actor but also on the fidelity of the world model’s representation. Effective explanations should, therefore, incorporate the learned dynamics of the environment.

In this work, we propose a method that leverages the imagination technique from the training process to generate stepwise, contrastive explanations during inference. Our approach systematically compares predicted states, actions, and value and reward estimates to evaluate the observed trajectory. This analysis provides insights into whether failures arise from inaccuracies in the world model, errors in value estimation, or deficiencies in reward prediction. We demonstrate the effectiveness of our method across multiple goal-oriented tasks.