Nestedness and modularity are measures of ecological networks whose causative effects are little understood. We analyzed antagonistic plant–herbivore bipartite networks using common gardens in two contrasting environments comprised of aspen trees with differing evolutionary histories of defence against herbivores. These networks were tightly connected owing to a high level of specialization of arthropod herbivores that spend a large proportion of the life cycle on aspen. The gardens were separated by ten degrees of latitude with resultant differences in abiotic conditions. We evaluated network metrics and reported similar connectance between gardens but greater numbers of links per species in the northern common garden. Interaction matrices revealed clear nestedness, indicating subsetting of the bipartite interactions into specialist divisions, in both the environmental and evolutionary aspen groups, although nestedness values were only significant in the northern garden. Variation in plant vulnerability, measured as the frequency of herbivore specialization in the aspen population, was significantly partitioned by environment (common garden) but not by evolutionary origin of the aspens. Significant values of modularity were observed in all network matrices. Trait-matching indicated that growth traits, leaf morphology, and phenolic metabolites affected modular structure in both the garden and evolutionary groups, whereas extra-floral nectaries had little influence. Further examination of module configuration revealed that plant vulnerability explained considerable variance in web structure. The contrasting conditions between the two gardens resulted in bottom-up effects of the environment, which most strongly influenced the overall network architecture, however, the aspen groups with dissimilar evolutionary history also showed contrasting degrees of nestedness and modularity. Our research therefore shows that, while evolution does affect the structure of aspen–herbivore bipartite networks, the role of environmental variations is a dominant constraint.