Hippocampal replay is a widely studied phenomenon wherein special neurons of the hippocampus encoding spacial locations – place cells – show a sequential reactivation during periods of immobility, often representing trajectories to or from reward locations, as observed in foraging rodents. Several computational models have been proposed to explain how this phenomenon could contribute to memory consolidation and action planning. However, certain aspects of the mechanism behind hippocampal replay remain unclear, such as why reactivation is biased towards both reward sites and decision points. Here, we propose that both expected reward (satisfying hunger) and expected information gain (satisfying curiosity) contribute to determine the priority of events to be replayed. To test this, we present the Epistemic Replay Algorithm (ERA), which bridges reinforcement learning and active inference into a single computational model. We evaluate the ERA in five experiments spanning three maze types: linear maze, non-stationary maze, double T-maze. Our results first showcase that more curious agents explore more thoroughly while they are still capable of exploiting optimal rewards; and they can adapt faster to changing environments. Further, we find that the ERA model accounts for a larger number of hippocampal replay properties compared to non-curious models, including (i) a broad-to-specific progression of hippocampal replay events; (ii) symmetric replay around decision points; and (iii) the preferential reactivation of both reward sites and decision points. We derive new predictions to further test the model and discuss its implications compared to alternative accounts.