Group formation and coordination are fundamental characteristics of living matter, essential for performing tasks and ensuring survival. Interactions between individuals play a key role in group formation, and the impact of resource distributions is a vibrant area of research. As of now, an understanding of how patchy resource distributions determine group dynamics is not yet fully understood. Studying active particles in controlled optical landscapes as energy sources, we demonstrate a non-monotonic dependency of group size on landscape patchiness, with the smallest groups forming when the patches match the active particles’ size. A similar relationship is observed in terms of group stability, evidenced by a reduced rate of individual exchange in patchy environments compared to homogeneous conditions. Reduced group sizes can be beneficial to optimise resources in heterogeneous environments and to control information flow within populations. Our results provide insights into the role of patchy landscapes and uneven energy distributions in active matter and hold implications for refining swarm intelligence algorithms, enhancing crowd management techniques, and tailoring colloidal self-assembly.