Background Three studies assessed the association of early life adversity (ELA) and hippocampal volumes in depressed patients, of which one was negative and the two others did not control for several potential confounding variables. Since the association of ELA and hippocampal volumes differ in male and female healthy volunteers, we investigated the association of ELA and hippocampal volumes in depressed patients, while focusing specifically on sex and controlling for several relevant socio-demographic and clinical variables. Methods Sixty-three depressed in-patients treated in a psychiatric setting, with a current Major Depressive Episode (MDE) and a Major Depressive Disorder (MDD) were included and assessed for ELA. Hippocampal volumes were measured with brain magnetic resonance imaging (MRI) and automatic segmentation. They were compared between patients with (n = 28) or without (n = 35) ELA. After bivariate analyses, multivariate regression analyses tested the interaction of sex and ELA on hippocampal volume and were adjusted for several potential confounding variables. The subgroups of men (n = 26) and women (n = 37) were assessed separately. Results Patients with ELA had a smaller hippocampus than those without ELA (4.65 (±1.11) cm3 versus 5.25 (±1.01) cm3), bivariate: p = 0.03, multivariate: HR = 0.40, 95%CI [0.23;0.71], p = 0.002), independently from other factors. This association was found in men (4.43 (±1.22) versus 5.67 (±0.77) cm3), bivariate: p = 0.006, multivariate HR = 0.23, 95%CI [0.06;0.82], p = 0.03) but not in women. Conclusion ELA is associated with a smaller hippocampus in male but not female depressed in-patients. The reasons for this association should be investigated in further studies.