Learning maximum entropy models from finite-size data sets: A fast data-driven algorithm allows sampling from the posterior distribution - Sorbonne Université Access content directly
Journal Articles Physical Review E Year : 2016

Learning maximum entropy models from finite-size data sets: A fast data-driven algorithm allows sampling from the posterior distribution

Ulisse Ferrari

Abstract

Maximum entropy models provide the least constrained probability distributions that reproduce statistical properties of experimental datasets. In this work we characterize the learning dynamics that maximizes the log-likelihood in the case of large but finite datasets. We first show how the steepest descent dynamics is not optimal as it is slowed down by the inhomogeneous curvature of the model parameters' space. We then provide a way for rectifying this space which relies only on dataset properties and does not require large computational efforts. We conclude by solving the long-time limit of the parameters' dynamics including the randomness generated by the systematic use of Gibbs sampling. In this stochastic framework, rather than converging to a fixed point, the dynamics reaches a stationary distribution, which for the rectified dynamics reproduces the posterior distribution of the parameters. We sum up all these insights in a “rectified” data-driven algorithm that is fast and by sampling from the parameters' posterior avoids both under- and overfitting along all the directions of the parameters' space. Through the learning of pairwise Ising models from the recording of a large population of retina neurons, we show how our algorithm outperforms the steepest descent method.
Fichier principal
Vignette du fichier
1507.04254.pdf (1.46 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-01379105 , version 1 (15-12-2022)

Identifiers

Cite

Ulisse Ferrari. Learning maximum entropy models from finite-size data sets: A fast data-driven algorithm allows sampling from the posterior distribution. Physical Review E , 2016, 94 (2), pp.023301. ⟨10.1103/PhysRevE.94.023301⟩. ⟨hal-01379105⟩
37 View
24 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More