Recently, several deep learning models have been pro-posed for 3D human pose estimation. Nevertheless, mostof these approaches only focus on the single-person caseor estimate 3D pose of a few people at high resolution.Furthermore, many applications such as autonomous driv-ing or crowd analysis require pose estimation of a largenumber of people possibly at low-resolution. In this work,we present PandaNet (Pose estimAtioN and DectectionAnchor-based Network), a new single-shot, anchor-basedand multi-person 3D pose estimation approach. The pro-posed model performs bounding box detection and, for eachdetected person, 2D and 3D pose regression into a singleforward pass. It does not need any post-processing to re-group joints since the network predicts a full 3D pose foreach bounding box and allows the pose estimation of a pos-sibly large number of people at low resolution. To managepeople overlapping, we introduce a Pose-Aware Anchor Se-lection strategy. Moreover, as imbalance exists between dif-ferent people sizes in the image, and joints coordinates have ifferent uncertainties depending on these sizes, we pro-pose a method to automatically optimize weights associatedto different people scales and joints for efficient training.PandaNet surpasses previous single-shot methods on sev-eral challenging datasets: a multi-person urban virtual butvery realistic dataset (JTA Dataset), and two real world 3Dmulti-person datasets (CMU Panoptic and MuPoTS-3D).