Balancing Diversity and Focus in Exploration with Dynamic State Clustering

Exploration remains a fundamental challenge in reinforcement learning (RL), especially in environments with sparse, deceptive, or delayed rewards. Existing methods often rely on intrinsic rewards or goal-directed strategies, but many struggle to reliably identify informative exploratory targets in complex or high-dimensional settings. We propose Dynamic State Clustering (DSC), a simple yet effective approach that dynamically clusters the state space at the frontier of exploration. The core idea is to bias the training of a clustering model using exploration value estimates, ensuring that the resulting cluster centers represent states with high potential for discovery. Goals are sampled directly from these centers, eliminating the need for separate ranking or filtering stages. This design enables the agent to adaptively balance diverse exploration and focused search, depending on the evolving structure of the environment. We evaluate DSC across several diverse domains, including Montezuma’s Revenge, a canonical benchmark for long-horizon exploration, and show that it consistently matches or outperforms state-of-the-art methods while demonstrating strong robustness across tasks. ... mehrAdditionally, we illustrate the interpretability of our method through visualizations of the evolving cluster structure, providing insight into the agent’s decision-making process during exploration.