Members of the diverse bacterial phylum Bacteroidetes have colonized virtually all types of habitats on Earth. They are among the major members of the microbiota of animals, especially in the gastrointestinal tract, can act as pathogens and are frequently found in soils, oceans and freshwater. In these contrasting ecological niches, Bacteroidetes are increasingly regarded as specialists for the degradation of high molecular weight organic matter, i.e., proteins and carbohydrates. This review presents the current knowledge on the role and mechanisms of polysaccharide degradation by Bacteroidetes in their respective habitats. The recent sequencing of Bacteroidetes genomes confirms the presence of numerous carbohydrate-active enzymes covering a large spectrum of substrates from plant, algal, and animal origin. Comparative genomics reveal specific Polysaccharide Utilization Loci shared between distantly related members of the phylum, either in environmental or gut-associated species. Moreover, Bacteroidetes genomes appear to be highly plastic and frequently reorganized through genetic rearrangements, gene duplications and lateral gene transfers (LGT), a feature that could have driven their adaptation to distinct ecological niches. Evidence is accumulating that the nature of the diet shapes the composition of the intestinal microbiota. We address the potential links between gut and environmental bacteria through food consumption. LGT can provide gut bacteria with original sets of utensils to degrade otherwise refractory substrates found in the diet. A more complete understanding of the genetic gateways between food-associated environmental species and intestinal microbial communities sheds new light on the origin and evolution of Bacteroidetes as animals' symbionts. It also raises the question as to how the consumption of increasingly hygienic and processed food deprives our microbiota from useful environmental genes and possibly affects our health.