Manual dexterity measures can be useful for early detection of age-related functional decline and for prediction of cognitive decline. However, what aspects of sensorimotor function to assess remains unclear. Manual dexterity markers should be able to separate impairments related to cognitive decline from those related to healthy aging. In this pilot study, we aimed to compare manual dexterity components in patients diagnosed with cognitive decline (mean age: 84 years, N = 11) and in age comparable cognitively intact elderly subjects (mean age: 78 years, N = 11). In order to separate impairments due to healthy aging from deficits due to cognitive decline we also included two groups of healthy young adults (mean age: 26 years, N = 10) and middle-aged adults (mean age: 41 years, N = 8). A comprehensive quantitative evaluation of manual dexterity was performed using three tasks: (i) visuomotor force tracking, (ii) isochronous single finger tapping with auditory cues, and (iii) visuomotor multi-finger tapping. Results showed a highly significant increase in force tracking error with increasing age. Subjects with cognitive decline had increased finger tapping variability and reduced ability to select the correct tapping fingers in the multi-finger tapping task compared to cognitively intact elderly subjects. Cognitively intact elderly subjects and those with cognitive decline had prolonged force release and reduced independence of finger movements compared to young adults and middle-aged adults. The findings suggest two different patterns of impaired manual dexterity: one related to cognitive decline and another related to healthy aging. Manual dexterity tasks requiring updating of performance, in accordance with (temporal or spatial) task rules maintained in short-term memory, are particularly affected in cognitive decline. Conversely, tasks requiring online matching of motor output to sensory cues were affected by age, not by cognitive status. Remarkably, no motor impairments were detected in patients with cognitive decline using clinical scales of hand function. The findings may have consequences for the development of manual dexterity markers of cognitive decline.