Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Selection-Expansion: A Unifying Framework for Motion-Planning and Diversity Search Algorithms

Abstract : Reinforcement learning agents need a reward signal to learn successful policies. When this signal is sparse or the corresponding gradient is deceptive, such agents need a dedicated mechanism to efficiently explore their search space without relying on the reward. Looking for a large diversity of behaviors or using Motion Planning (MP) algorithms are two options in this context. In this paper, we build on the common roots between these two options to investigate the properties of two diversity search algorithms, the Novelty Search and the Goal Exploration Process algorithms. These algorithms look for diversity in an outcome space or behavioral space which is generally hand-designed to represent what matters for a given task. The relation to MP algorithms reveals that the smoothness, or lack of smoothness of the mapping between the policy parameter space and the outcome space plays a key role in the search efficiency. In particular, we show empirically that, if the mapping is smooth enough, i.e. if two close policies in the parameter space lead to similar outcomes, then diversity algorithms tend to inherit exploration properties of MP algorithms. By contrast, if it is not, diversity algorithms lose these properties and their performance strongly depends on specific heuristics, notably filtering mechanisms that discard some of the explored policies.
Document type :
Preprints, Working Papers, ...
Complete list of metadata

https://hal.sorbonne-universite.fr/hal-03196479
Contributor : Alexandre Chenu <>
Submitted on : Monday, April 12, 2021 - 7:32:55 PM
Last modification on : Wednesday, May 19, 2021 - 11:58:08 AM
Long-term archiving on: : Tuesday, July 13, 2021 - 7:15:10 PM

File

Selection_Expansion_HAL.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03196479, version 1

Citation

Alexandre Chenu, Nicolas Perrin, Stephane Doncieux, Olivier Sigaud. Selection-Expansion: A Unifying Framework for Motion-Planning and Diversity Search Algorithms. 2021. ⟨hal-03196479⟩

Share

Metrics

Record views

55

Files downloads

27