In this paper we study the $\mathbb{C}^*$-fixed points in moduli spaces of Higgs bundles over a compact Riemann surface for a complex semisimple Lie group and its real forms. These fixed points are called Hodge bundles and correspond to complex variations of Hodge structure. We introduce a topological invariant for Hodge bundles that generalizes the Toledo invariant appearing for Hermitian Lie groups. A main result of this paper is a bound on this invariant which generalizes both the Milnor-Wood inequality of the Hermitian case and the Arakelov inequalities of classical variations of Hodge structure. When the generalized Toledo invariant is maximal, we establish rigidity results for the associated variations of Hodge structure which generalize known rigidity results for maximal Higgs bundles and their associated maximal representations in the Hermitian case.