For augmented reality experiences, users wear head-mounted displays (HMD) while listening to real and virtual sound sources. This paper assesses the impact of wearing an HMD on localization accuracy of real sources. Eighteen blindfolded participants completed a localization task on 32 loudspeakers while wearing either no HMD, a bulky visor HMD, or a glass visor HMD. Results demonstrate that the HMDs had a significantly impact on participants’ localization performance, increasing local great circle angle error by 0.9°, and that the glass visor HMD demonstrably increased the rate of up–down confusions in the responses by 0.9–1.1%. These results suggest that wearing an HMD has a sufficiently small impact on real source localization that it can safely be considered as an HMD-free condition in most but the most demanding AR auditory localization studies.