Tightly focused optical dipole traps induce vector light shifts (" fictitious magnetic fields ") which complicate their use for single-atom trapping and manipulation. The problem can be mitigated by adding a larger, real magnetic field, but this solution is not always applicable; in particular, it precludes fast switching to a field-free configuration. Here we show that this issue can be addressed elegantly by deliberately adding a small elliptical polarization component to the dipole trap beam. In our experiments with single 87 Rb atoms laser-cooled in a chopped trap, we observe improvements up to a factor 11 of the trap lifetime compared to the standard, seemingly ideal linear polarization. This effect results from a modification of heating processes via spin-state diffusion in state-dependent trapping potentials. We develop Monte-Carlo simulations of the evolution of the atom's internal and motional states and find that they agree quantitatively with the experimental data. The method is general and can be applied in all experiments where the longitudinal polarization component is non-negligible.