Within-colony phenotypic diversity can play an essential role in some eusocial insect taxa by increasing the performance of division of labor, thereby increasing colony fitness. Empirical studies of the effect of phenotypic diversity on colony fitness mostly focused on species with discrete castes (workers, soldiers) or with continuously and highly morphologically variable workers, which is not the most common case. Indeed, most species exhibit continuous but limited worker morphological variation. It is still unclear whether this variation impacts colony fitness. To test this, we reduced the worker size diversity in 25 colonies of the ant Temnothorax nylanderi and compared their performances to 25 control colonies. We reared these colonies in the laboratory and measured the effect of treatment (reduced diversity or control) and colony size (number of workers) on colony performance at six challenges, as well as on worker mortality and brood production. The reduction of worker size diversity did not affect colony performance nor mortality and brood production. As expected, colony performance and brood production increased with colony size. These results suggest that worker size diversity may not be under positive selection in this species, but rather the product of a lack of developmental canalization. We propose that social life could decrease the selective pressures maintaining developmental canalization, subsequently leading to higher size diversity without necessarily increasing colony performance.