The cancer stem cell hypothesis poses that the bulk of differentiated cells are non-tumorigenic and only a subset of cells with self-renewal capabilities drive tumor initiation and progression. This means that differentiation could have a tumor-suppressive effect. Accumulating evidence shows, however, that in some solid tumors, like colorectal cancer, such a hierarchical organization is necessary. The identification of Lgr5 as a reliable marker of normal intestinal epithelial stem cells, together with strategies to trace cell lineages within tumors and the possibility to selectively ablate these cells, have proven the relevance of Lgr5+ cells for cancer progression. On the contrary, the role of Lgr5− cells during this process remains largely unknown. In this review, we explore available evidence pointing towards possible selective advantages of cancer cells organized hierarchically and its resulting cell heterogeneity. Clear evidence of plasticity between cell states, in which loss of Lgr5+ cells can be replenished by dedifferentiation of Lgr5− cells, shows that cell hierarchies could grant adaptive traits to tumors upon changing selective pressures, including those derived from anticancer therapy, as well as during tumor progression to metastasis.