Background: The outbreak of the COVID-19 epidemic lead to high levels of morbidity and mortality around the globe. Consequences of this outbreak and possible associated infection are an increase in mental health disorders and an increased likelihood of internalizing problems, particularly depression. However, to date few studies have tested this hypothesis while taking into account individuals’ preexisting mental health difficulties. Methods: We used longitudinal data collected among 729 persons in the context of the French TEMPO cohort between March and June 2020 (7 waves of data collection). COVID-19-like symptoms as well as anxiety/depression (assessed by the Adult Self Report), were reported at each wave of data collection. To study the relationship between COVID-19-like symptoms and anxiety/depression, we used generalized estimation equation (GEE) models controlled for socio-demographic and health-related characteristics, including anxiety/depression prior to 2020. Results: Overall, 27.2% of study participants reported anxiety/depression during lockdown. 17.1% of participants reported COVID-19-like symptoms during the course of follow-up, 7.3% after the beginning of lockdown, with an average number of 2.7 symptoms, and 3.6% reported respiratory distress. In multivariate analyses, nearly all the considered indicators of COVID-19-like symptoms were associated with higher odds of symptoms of anxiety/depression (symptoms Yes/No: OR = 1.66, 95% CI = 1.08–2.55; symptoms after the beginning of lockdown: OR = 1.91, 95% CI = 1.03–3.52; number of symptoms: OR for each additional symptom = 1.19, 95% CI = 1.02–1.39. This relationship exists after taking into account prior symptoms of anxiety/depression, which are associated with a 5-fold increased likelihood of psychological distress. And this impact is stronger among men than women. Conclusions: Our study shows higher risk of anxiety/depression among persons who experienced COVID-19-like symptoms, even after accounting for prior mental health difficulties. COVID-19 infection could have both a direct and indirect impact on the occurrence of psychological difficulties, and this association should be studied in greater detail.