We investigate the linear stability of inviscid columnar vortices with respect to finite energy perturbations. For a large class of vortex profiles, we show that the linearized evolution group has a sub-exponential growth in time, which means that the associated growth bound is equal to zero. This implies in particular that the spectrum of the linearized operator is entirely contained in the imaginary axis. This contribution complements the results of our previous work [10], where spectral stability was established for the linearized operator in the enstrophy space.