Flaking is a well-known pattern on rich clay stone. As swelling of clay minerals may induce crack propagation under fatigue, a fracture mechanics approach is proposed to investigate its impact on such decay pattern. A clay-bearing sandstone from the Thüringen region is studied because of the scaling effect observed at its surface when exposed to environmental conditions. Semi-circular bending specimens adapted to stone heritage studies are prepared and three configurations are tested, in order to measure toughness with respect to the bedding of this sandstone. Deformations are measured during relative humidity variations. They are measured anisotropic due to position and orientation of the clay phases within the stone. The influence of such natural dimensional variations on Young modulus and fracture toughness is studied. It appears that the induced damage is oriented and is the consequence of opening of the initial microcracks in the direction perpendicular to the maximum swelling. This damage induces an evolution of the fracture properties and behaviour. Toughness decreases as relative humidity increases depending on the orientation of the microcracking. Moreover, the toughness anisotropy of this sandstone appears during humidification. After several cycles of swelling, the microcracking induces an increase in toughness when notch is perpendicular to them. This may explain some stone deterioration patterns, as flakes subparallel to the stone surface.