For fifteen years now, visual tracking has been a very active research area of the computer vision community. But an increasing amount of works can be observed in the last five years. This has led to the development of numerous algorithms that can deal with more and more complex video sequences. Each of them has its own strengths and weaknesses. That is the reason why it becomes necessary to compare those algorithms. For this purpose, some datasets dedicated to visual tracking as well as, sometimes, their ground truth annotation files, are regularly made publicly available by researchers. However, each dataset has its own speci-ficities and is sometimes dedicated to test the ability of some algorithms to tackle only one or a few specific visual tracking subproblems. This article provides an overview of some of the datasets that are most used by the visual tracking community, but also of others that address specific tasks. We also propose a cartography of these datasets from a novel perspective, namely that of the difficulties the datasets present for visual tracking.