Massive multiple-input multiple-output (MIMO) systems operating at millimeter-wave (mmWave) frequencies promise to satisfy the demand for higher data rates in mobile communication networks. A practical challenge that arises is the calibration in amplitude and phase of these massive MIMO systems, as the antenna elements are too densely packed to provide a separate calibration branch for measuring them independently. Over-the-air (OTA) calibration methods are viable solutions to this problem. In contrast to previous works, the here presented OTA calibration method is investigated and optimized for subarray-based hybrid beamforming (SBHB) systems. SBHB systems represent an efficient architectural solution to realize massive MIMO systems. Moreover, based on OTA scattering parameter measurements, the ambiguities of the phase shifters are exploited and two criteria to optimize the beam pattern are formulated. Finally, the optimization criteria are examined in measurements utilizing a novel SBHB receiver system operating at 27.8 GHz.