Background: Persistent physical symptoms are common after a coronavirus disease 2019 (COVID-19) episode, but their pathophysiological mechanisms remain poorly understood. In this study, we aimed to explore the association between anxiety and depression at 1-month after acute infection and the presence of fatigue, dyspnea, and pain complaints at 3-month follow-up. Methods: We conducted a prospective study in patients previously hospitalized for COVID-19 followed up for 3 months. The Hospital Anxiety and Depression Scale (HAD-S) was administered by physicians at 1-month follow-up, and the presence of fatigue, dyspnea, and pain complaints was assessed at both 1 month and 3 months. Multivariable logistic regressions explored the association between anxiety and depression subscores and the persistence of each of the physical symptom at 3 months. Results: A total of 84 patients were included in this study (Median age: 60 years, interquartile range: 50.5–67.5 years, 23 women). We did not find any significant interaction between anxiety and the presence of fatigue, dyspnea, or pain complaints at 1 month in predicting the persistence of these symptoms at 3 months (all p ≥ 0.36). In contrast, depression significantly interacted with the presence of pain at 1 month in predicting the persistence of pain at 3 months (OR: 1.60, 95% CI: 1.02–2.51, p = 0.039), with a similar trend for dyspnea (OR: 1.51, 95% CI: 0.99–2.28, p = 0.052). Discussion and Conclusion: Contrary to anxiety, depression after an acute COVID-19 episode may be associated with and increased risk of some persistent physical symptoms, including pain and dyspnea.