Answers Unite! Unsupervised Metrics for Reinforced Summarization Models - Sorbonne Université Access content directly
Conference Papers Year : 2019

Answers Unite! Unsupervised Metrics for Reinforced Summarization Models

Sylvain Lamprier
Benjamin Piwowarski
Jacopo Staiano
  • Function : Author


Abstractive summarization approaches based on Reinforcement Learning (RL) have recently been proposed to overcome classical likelihood maximization. RL enables to consider complex, possibly non differentiable, metrics that globally assess the quality and relevance of the generated outputs. ROUGE, the most used summarization metric, is known to suffer from bias towards lexical similarity as well as from sub-optimal accounting for fluency and readability of the generated abstracts. We thus explore and propose alternative evaluation measures: the reported human-evaluation analysis shows that the proposed metrics, based on Question Answering, favorably compare to ROUGE – with the additional property of not requiring reference summaries. Training a RL-based model on these metrics leads to improvements (both in terms of human or automated metrics) over current approaches that use ROUGE as reward.
Fichier principal
Vignette du fichier
D19-1320.pdf (301.97 Ko) Télécharger le fichier
Origin : Publisher files allowed on an open archive

Dates and versions

hal-02350999 , version 1 (07-10-2021)



Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano. Answers Unite! Unsupervised Metrics for Reinforced Summarization Models. 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Nov 2019, Hong Kong, China. pp.3237-3247, ⟨10.18653/v1/D19-1320⟩. ⟨hal-02350999⟩
109 View
23 Download



Gmail Facebook Twitter LinkedIn More