Providing personalized point-of-interest (POI) recommendation has become a major issue with the rapid emergence of location-based social networks (LBSNs). Unlike traditional recommendation approaches, the LBSNs application domain comes with significant geographical and temporal dimensions. Moreover most of traditional recommendation algorithms fail to cope with the specific challenges implied by these two dimensions. Fusing geographical and temporal influences for better recommendation accuracy in LBSNs remains unexplored, as far as we know. We depict how matrix factorization can serve POI recommendation, and propose a novel attempt to integrate both geographical and temporal influences into matrix factorization. Specifically we present GeoMF-TD, an extension of geographical matrix factorization with temporal dependencies. Our experiments on a real dataset shows up to 20% benefit on recommendation precision.