Information can be perceived from a multiplicity of spatial perspectives, which is central to effectively understanding and interacting with our environment and other people. Sensory impairments such as blindness are known to impact spatial representations and perspective-taking is often thought of as a visual process. However, disturbed functioning of other sensory systems (e.g., vestibular, proprioceptive and auditory) can also influence spatial perspective-taking. These lines of research remain largely separate, yet together they may shed new light on the role that each sensory modality plays in this core cognitive ability. The findings to date reveal that spatial cognitive processes may be differently affected by various types of sensory loss. The visual system may be crucial for the development of efficient allocentric (object-to-object) representation; however, the role of vision in adopting another's spatial perspective remains unclear. On the other hand, the vestibular and the proprioceptive systems likely play an important role in anchoring the perceived self to the physical body, thus facilitating imagined self-rotations required to adopt another's spatial perspective. Findings regarding the influence of disturbed auditory functioning on perspective-taking are so far inconclusive and thus await further data. This review highlights that spatial perspective-taking is a highly plastic cognitive ability, as the brain is often able to compensate in the face of different sensory loss.