We develop a global setting for modeling thermo-visco-elastic materials that satisfy the principles of thermodynamics and are properly invariant. This setting encompasses many known solid and fluid models, as well as new models with internal variables that generalize the Maxwell rheological model. Complex fluid models such as the Oldroyd B model are shown to belong to the above general family of models. The specific Oldroyd B model is however found to be seriously lacking in terms of satisfying the second principle of thermodynamics. On the contrary, a second complex fluid model based on the Zaremba-Jaumann derivative is shown to satisfy the second principle of thermodynamics.