Under spatially incoherent illumination, time-domain full-field optical coherence tomography (FFOCT) offers the possibility to achieve in vivo retinal imaging at cellular resolution over a wide field of view. Such performance is possible, albeit there is the presence of ocular aberrations even without the use of classical adaptive optics. While the effect of aberrations in FFOCT has been debated these past years, mostly on low-order and static aberrations, we present, for the first time to our knowledge, a method enabling a quantitative study of the effect of statistically representative static and dynamic ocular aberrations on FFOCT image metrics, such as SNR, resolution, and image similarity. While we show that ocular aberrations can decrease FFOCT SNR and resolution by up to 14 dB and fivefold, we take advantage of such quantification to discuss different possible compromises between performance gain and adaptive optics complexity and speed, to optimize both sensor-based and sensorless FFOCT high-resolution retinal imaging.