The past decade has witnessed a rapid increase in the use of screen media in families, and infants are exposed to screens at younger ages than ever before. The objective of this review is twofold: (1) to understand the correlates and demographic factors determining exposure to screens, including interactive screens, when available, and (2) to study the effects of watching screens and using touchscreens on cognitive development, during the first 3 years of life. We argue that the effects of screen viewing depend mostly on contextual aspects of the viewing rather than on the quantity of viewing. That context includes the behavior of adult caregivers during viewing, the watched content in relation to the child's age, the interactivity of the screen and whether the screen is in the background or not. Depending on the context, screen viewing can have positive, neutral or negative effects on infants' cognition.