Background: In France, with the growing scarcity of gynecologists and a globally low and socially differentiated coverage of cervical cancer screening (CCS), general practitioners (GPs) are valuable resources to improve screening services for women. Still all GPs do not perform Pap smears. In order to promote this screening among GPs, the characteristics of physicians who never perform CCS should be more precisely specified. Besides already-known individual characteristics, the contextual aspects of the physicians’ office, such as gynecologist density in the area, could shape GPs gynecological activities. Methods: To analyze county (département) characteristics of GPs’ office associated with no performance of CCS, we used a representative sample of 1063 French GPs conducted in 2009 and we constructed mixed models with two levels, GP and county. Results: Almost 35% (n = 369) of the GPs declared never performing CCS. GPs working in counties with a poor GP-density per inhabitants were more likely to perform CCS (odds ratio (OR) = 0.52 for each increase of density by 1 GP per 10,000 inhabitants, 95% confidence interval (CI) = 0.37–0.74). On the contrary, GPs working in counties with an easier access to a gynecologist were more likely not to perform CCS (OR = 1.06 for each increase of density by 1 gynecologist per 100,000 women, 95%CI = 1.03–1.10 and OR = 2.02 if the first gynecologist is reachable in less than 15 min, 95%CI = 1.20–3.41) as well as GPs working in areas with a poverty rate above the national average (OR = 1.66, 95%CI = 1.09–2.54). These contextual characteristics explain most of the differences between counties concerning rates of not performing CCS. Conclusions: Specific programs should be developed for GPs working in contexts unfavorable to their involvement in CCS.