We investigate numerically and theoretically the precursory intermittent activity characterizing the preliminary phase of damage accumulation prior to failure of quasi-brittle solids. We use a minimal but thermodynamically consistent model of damage growth and localization developed by Berthier et al. (2017). The approach accounts for both microstructural disorder and non-local interactions and permits inferring a complete scaling description of the spatio-temporal structure of failure precursors. By developing a theoretical model of damage growth in disordered elasto-damageable specimen, we demonstrate that these scaling relations emerge from the physics of elastic manifolds driven in disordered media, while the divergence of these quantities close to failure is reminiscent of the loss of stability of the specimen at the localization threshold. Our study sorts out a long-standing debate on the nature of the compressive failure point and the origin of the universal statistics of the precursors preceding it. Our analysis rules out a critical-point scenario in which the divergence of the precursor size close to failure is signature of a second-order phase transition governed by the microstructural disorder. Instead, we show that while the jerky evolution of damage prior to failure results from the presence of material disorder, the latter does not significantly change the nature of the localization process, which is an instability well described by standard bifurcation theory of homogeneous systems. Finally, we harness our detailed understanding of the precursory statistics to design a methodology to estimate the residual lifetime of a structure from the statistical analysis of precursors. This method relevant for structural health monitoring is shown to perform rather accurately on our data.