Maxwell's and the Drude-Born-Fedorov equations model EM-waves in chiral media. The Factorization method is generalized for inverse scattering by chiral objects. The case of a homogeneous chiral sphere is done analytically. Scattering by a chiral cylinder motivates the Factorization method for the vector Helmholtz equation. Numerical examples illustrate the theoretical results. Scattering from periodic chiral structures is another possible application of the generalized Factorization method.