The Mediterranean Sea is a miniature ocean divided by the Sicily Strait into two basins with a marked west to east trophic gradient and separated of the nearby eastern Atlantic Ocean by the Strait of Gibraltar. Here, we test the hypothesis that these physical and environmental barriers favor the development of specific prokaryotic assemblages, leading to changes in community structure both in the vertical and horizontal spatial scales. By analyzing taxonomic and phylogenetic diversity using amplicon sequence variants (ASVs) of the 16S rRNA gene, we show that there is indeed marked vertical segregation of prokaryotic groups, similar to that found in other areas of the ocean, but also a clear horizontal structuring among the two Mediterranean basins and the adjacent Atlantic waters. Prokaryotic diversity increased with depth and toward the Atlantic, whereas the easternmost stations displayed more phylogenetically diverse phylotypes, despite harboring globally less diverse communities. Basin-indicator taxa (ASVs) accounted for a large fraction of the community (between 50% and 80%) in each of the basins at the surface and bathypelagic layers, being associated with different environmental variables. The existence of biogeographic and environmental barriers in the Mediterranean Sea is likely related to the trophic gradient at the surface and the isolation of water bodies in depth due to the Gibraltar and Sicily straits. Our work highlights the importance of studying microbial regional biogeography and provides the basis for future studies on the impact of this regionalization in the function of Mediterranean Sea prokaryotic communities.