Across all species, retinal ganglion cells (RGCs) are the first retinal neurons generated during development, followed by the other retinal cell types. How are retinal progenitor cells (RPCs) able to produce these cell types in a specific and timely order? Here, we will review the different models of retinal neurogenesis proposed over the last decades as well as the extrinsic and intrinsic factors controlling it. We will then focus on the molecular mechanisms, especially the cascade of transcription factors that regulate, more specifically, RGC fate. We will also comment on the recent discovery that the ciliary marginal zone is a new stem cell niche in mice contributing to retinal neurogenesis, especially to the generation of ipsilateral RGCs. Furthermore, RGCs are composed of many different subtypes that are anatomically, physiologically, functionally, and molecularly defined. We will summarize the different classifications of RGC subtypes and will recapitulate the specification of some of them and describe how a genetic disease such as albinism affects neurogenesis, resulting in profound visual deficits.