Infertility is becoming a major public health problem, with increasing frequency due to medical, environmental and societal causes. The increasingly late age of childbearing, growing exposure to endocrine disruptors and other reprotoxic products, and increasing number of medical reproductive dysfunctions (endometriosis, polycystic ovary syndrome, etc.) are among the most common causes. Fertility relies on fine-tuned control of both neuroendocrine function and reproductive behaviors, those are critically regulated by sex steroid hormones. Testosterone and estradiol exert organizational and activational effects throughout life to establish and activate the neural circuits underlying reproductive function. This regulation is mediated through estrogen receptors (ERs) and androgen receptor (AR). Estradiol acts mainly via nuclear estrogen receptors ERα and ERβ. The aim of this review is to summarize the genetic studies that have been undertaken to comprehend the specific contribution of ERα and ERβ in the neural circuits underlying the regulation of the hypothalamic-pituitary-gonadal axis and the expression of reproductive behaviors, including sexual and parental behavior. Particular emphasis will be placed on the neural role of these receptors and the underlying sex differences.