We used a homodyne detection to investigate the gyration of magnetic vortex cores in Fe islands on W(110) with spin-polarized scanning tunneling microscopy at liquid helium temperatures. The technique aims at local detection of the spin precession as a function of frequency using a radio-frequency (rf) modulation of the tunneling bias voltage. The gyration was excited by the resulting spin-polarized rf current in the tunneling junction. A theoretical analysis of different contributions to the frequency-dependent signals expected in this technique is given. These include, besides the ferromagnetic resonance signal, also signals caused by the non-linearity of the I(U) characteristics. The vortex gyration was modeled with micromagnetic finite element methods using realistic parameters for the tunneling current, its spin polarization, and the island shape, and simulations were compared with the experimental results. The observed signals are presented and critically analyzed.