The outcome of adult cochlear implantation is predicted positively by the involvement of visual cortex in speech processing, and negatively by the cross-modal recruitment of the right temporal cortex during and after deafness. How these two neurofunctional predictors concur to modulate cochlear implant (CI) performance remains unclear. In this fMRI study, we explore the joint involvement of occipital and right hemisphere regions in a visual-based phonological task in post-lingual deafness. Intriguingly, we show that some deaf subjects perform faster than controls. This behavioural effect is associated with reorganized connectivity across bilateral visual, right temporal and left inferior frontal cortices, but with poor CI outcome. Conversely, preserved normal-range reaction times are associated with left-lateralized phonological processing and good CI outcome. These results suggest that following deafness, involvement of visual cortex in the context of reorganized right-lateralized phonological processing compromises its availability for audiovisual synergy during adaptation to CI.