Trans-SAC: Robust and Transferable Maximum Entropy Reinforcement Learning for Heat Pump Control

Residential heating electrification through heat pumps is a cornerstone of building decarbonization, yet their potential for grid flexibility remains underutilized due to the difficulty of scaling control strategies across heterogeneous building stocks. Although reinforcement learning (RL) offers model-free adaptability, standard approaches often fail to scale, suffering from brittle policies that do not generalize across diverse thermal dynamics or discrete control actions that damage hardware. To address this, we introduce Trans-SAC (Transferable Soft Actor-Critic), a robust control framework designed to solve the “cold start” problem in city-scale deployments. By leveraging maximum entropy RL, Trans-SAC optimizes a dynamic trade-off between reward and entropy, ensuring continuous, hardware-safe actuation. Unlike prior work limited to fixed temperature bands, we target a challenging time-varying comfort objective under dynamic pricing and rigorously evaluate SAC against model predictive control (MPC) baselines, deep Q-networks (DQN) and proximal policy optimization (PPO), across a dataset of ten heterogeneous residential buildings calibrated with real-world weather and price data. ... mehrOur experiments reveal that while the standard feature-freezing technique accelerates training, they remain suboptimal due to physical mismatches in building envelopes, and independent learning remains prohibitively sample-inefficient. In contrast, Trans-SAC’s full-network adaptation strategy successfully bridges this gap, enabling robust generalization and reducing the training time and significantly outperforming other transfer baselines. This establishes Trans-SAC as a viable and data-efficient blueprint for aggregating large-scale heat pump fleets into smart buildings.