The Fokker action governing the motion of compact binary systems without spins is derived in harmonic coordinates at the fourth post-Newtonian approximation (4PN) of general relativity. Dimensional regularization is used for treating the local ultraviolet (UV) divergences associated with point particles, followed by a renormalization of the poles into a redefinition of the trajectories of the point masses. Effects at the 4PN order associated with wave tails propagating at infinity are included consistently at the level of the action. A finite part procedure based on analytic continuation deals with the infrared (IR) divergencies at spatial infinity, which are shown to be fully consistent with the presence of near-zone tails. Our end result at 4PN order is Lorentz invariant and has the correct self-force limit for the energy of circular orbits. However, we find that it differs from the recently published result derived within the ADM Hamiltonian formulation of general relativity [T. Damour, P. Jaranowski, and G. Schäfer, Phys. Rev. D 89, 064058 (2014)]. More work is needed to understand this discrepancy.