The current tsunami of deep learning has already conquered new areas, such as the generation of creative musical content. The motivation is in using the capacity of modern deep learning architectures and associated training and generation techniques to automatically learn musical styles from arbitrary musical corpora and then to generate musical samples from the estimated distribution, with some degree of control over the generation. In this article, we analyze the use of autoencoder architectures and how their ability for compressing information turns out to be an interesting source for generation. Autoencoders are good at representation learning, that is at extracting a compressed and abstract representation (a set of latent variables) common to the set of training examples. By choosing various instances of this abstract representation (i.e., by sampling the latent variables), one may efficiently generate various instances within the style which has been learnt. Furthermore, one may use more sophisticated ways of controlling the generation, like interpolation, recursion, or objective optimization, as will be illustrated by various examples.