Most animals, from humans to invertebrates, possess an ability to estimate numbers. This evolutionary advantage facilitates animals' choice of environments with more food sources, more conspecifics to increase mating success, and/or reduced predation risk among others. However, how the brain processes numerical information remains largely unknown. There are currently two lines of research interested in how numerosity of visual objects is perceived and analyzed in the brain. The first argues that numerosity is an advanced cognitive ability processed in highorder brain areas, while the second proposes that "numbers" are attributes of the visual scene and thus numerosity is processed in the visual sensory system. Recent evidence points to a sensory involvement in estimating magnitudes. In this Perspective, we highlight this evidence in two highly evolutionary distant species: humans and flies. We also discuss the advantages of studying numerical processing in fruit flies in order to dissect the neural circuits involved in and required for numerical processing. Based on experimental manipulation and the fly connectome, we propose a plausible neural network for number sense in invertebrates.