The Higgs sector of the Next-to-Minimal Two-Higgs-Doublet Model (N2HDM) is obtained from the Two-Higgs-Doublet Model (2HDM) containing two complex Higgs doublets, by adding a real singlet field. In this paper, we analyse the vacuum structure of the N2HDM with respect to the possibility of vacuum instabilities. We show that while one type of charge-and CP-preserving vacuum cannot coexist with deeper charge or CP-breaking minima, there is another type of vacuum whose stability is endangered by the possible occurrence of deeper charge-and CP-breaking minima. Analytical expressions relating the depth of different vacua are deduced. Parameter scans of the model are carried out that illustrate the regions of parameter space where the vacuum is either stable or metastable as well as the regions where tunnelling to deeper vacua gives rise to a too short lifetime of the vacuum. Taking other experimental and theoretical constraints into account, we find that the vacuum stability constraints have an important impact on the phenomenology of the N2HDM.