Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching - Sorbonne Université
Poster De Conférence Année : 2022

Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching

Pierre-Alexandre Kamienny
  • Fonction : Auteur
  • PersonId : 1189089
  • IdRef : 274575434
Jean Tarbouriech
  • Fonction : Auteur
  • PersonId : 1105513
Sylvain Lamprier
Alessandro Lazaric
  • Fonction : Auteur
  • PersonId : 1105515

Résumé

Learning meaningful behaviors in the absence of reward is a difficult problem in reinforcement learning. A desirable and challenging unsupervised objective is to learn a set of diverse skills that provide a thorough coverage of the state space while being directed, i.e., reliably reaching distinct regions of the environment. In this paper, we build on the mutual information framework for skill discovery and introduce UPSIDE, which addresses the coverage-directedness trade-off in the following ways: 1) We design policies with a decoupled structure of a directed skill, trained to reach a specific region, followed by a diffusing part that induces a local coverage. 2) We optimize policies by maximizing their number under the constraint that each of them reaches distinct regions of the environment (i.e., they are sufficiently discriminable) and prove that this serves as a lower bound to the original mutual information objective. 3) Finally, we compose the learned directed skills into a growing tree that adaptively covers the environment. We illustrate in several navigation and control environments how the skills learned by UPSIDE solve sparse-reward downstream tasks better than existing baselines.
Fichier principal
Vignette du fichier
3261_direct_then_diffuse_incrementa.pdf (5.16 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03923264 , version 1 (04-01-2023)

Identifiants

  • HAL Id : hal-03923264 , version 1

Citer

Pierre-Alexandre Kamienny, Jean Tarbouriech, Sylvain Lamprier, Alessandro Lazaric, Ludovic Denoyer. Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching. ICLR 2022, Apr 2022, Virtual, United States. 2022. ⟨hal-03923264⟩
17 Consultations
50 Téléchargements

Partager

More