Humans are capable of extracting spatial information through their sense of touch: when someone strokes their hand, they can easily determine stroke direction without visual information. However, when it comes to the coordinate system used to assign the spatial relations to the stimulation, it remains poorly understood how the brain selects the appropriate system for passive touch. In the study reported here, we investigated whether hand orientation can determine coordinate assignment to ambiguous tactile patterns, whether observers can cognitively override any orientation-driven perspectives on touch, and whether the adaptation transfers across body surfaces. Our results demonstrated that the orientation of the hand in the vertical plane determines the perspective taken: an external perspective is adopted when the hand faces the observer and a gaze-centred perspective is selected when the hand faces away. Participants were then adapted to a mirror-reversed perspective through training, and the results revealed that this adapted perspective holds for the adapted surface and generalises to non-adapted surfaces, including across the body midline. These results reveal plasticity in perspective taking which relies on low-level postural cues (hand orientation) but also on higher-order somatosensory processing that can override the low-level cues.