Fundamental limits of fiber link are set by non-reciprocal effects that violate the hypothesis of equality between forward and backward path. Non-reciprocal noise arises technically from the set-up asymmetry, and fundamentally by the Sagnac effect when the fiber link encloses a non-zero area. As a pre-requisite for observation of Sagnac effect in fiber links, we present a study on phase noise and frequency stability contributions affecting coherent optical frequency transfer in bi-directional fiber links. Both technical and fundamental limitations of Two-Way optical frequency transfer are discussed. Our model predicts and our experiments substantially verify that the dominant noise mechanism at low Fourier frequencies is the polarization asymmetry induced by the temperature and relative humidity variations impacted on fiber links. The flicker noise floor due to the non-reciprocal noise arising from polarization mode dispersion is evidenced for the first time. We perform a post-processing approach which enables us to remove this polarization noise, improve the long-term stability and remove a frequency bias. We evaluate the uncertainty contributions of all the effects discussed for our 50 km spooled fiber link, dominated by its non-reciprocal noise induced by polarization mode dispersion with uncertainty of 1.9( ± 0.8)( ± 1.2) × 10−20. After correction, the linear drift of the residual phase is as low as 27 yoctosecond/s, leading to an uncertainty of the frequency transfer of 2.6 ( ± 39) × 10−22, confirming its potential for searching for more fundamental effects such as Sagnac effect or transient frequency variation due to dark matter.