In this paper, we consider the general perturbations of piecewise Hamiltonian systems. A formula for the second order Melnikov functions is derived when the first order Melnikov functions vanish. As an application, we can improve an upper bound of the number of bifurcated limit cycles of a piecewise Hamiltonian system with quadratic polynomial perturbations.