Background: Patient-ventilator synchronization during non-invasive ventilation (NIV) can be assessed by visual inspection of flow and pressure waveforms but it remains time consuming and there is a large inter-rater variability, even among expert physicians. SyncSmart ™ software developed by Breas Medical (Mölnycke, Sweden) provides an automatic detection and scoring of patient-ventilator asynchrony to help physicians in their daily clinical practice. This study was designed to assess performance of the automatic scoring by the SyncSmart software using expert clinicians as a reference in patient with chronic respiratory failure receiving NIV. Methods: From nine patients, 20 min data sets were analyzed automatically by SyncSmart software and reviewed by nine expert physicians who were asked to score auto-triggering (AT), double-triggering (DT), and ineffective efforts (IE). The study procedure was similar to the one commonly used for validating the automatic sleep scoring technique. For each patient, the asynchrony index was computed by automatic scoring and each expert, respectively. Considering successively each expert scoring as a reference, sensitivity, specificity, positive predictive value (PPV), κ-coefficients, and agreement were calculated. Results: The asynchrony index assessed by SynSmart was not significantly different from the one assessed by the experts (18.9 ± 17.7 vs. 12.8 ± 9.4, p = 0.19). When compared to an expert, the sensitivity and specificity provided by SyncSmart for DT, AT, and IE were significantly greater than those provided by an expert when compared to another expert. Conclusions: SyncSmart software is able to score asynchrony events within the inter-rater variability. When the breathing frequency is not too high (<24), it therefore provides a reliable assessment of patient-ventilator asynchrony; AT is over detected otherwise.