The response to light of macromolecules in aqueous media is a significant concern for the design of complex fluids, and (bio) materials imparted with stimuli-triggered properties. It is common knowledge that polymer chains containing photochrome groups can be tailored to undergo a transition between poor and good solvent conditions upon exposure to light, thanks to the photoswitch of the polarity/hydrophobicity of their photochromes. In water, marked photo-responses are typically achieved in the vicinity of the low critical solution temperature (LCST) of polymers. To understand and optimize the variation of the LCST of amphiphilic polymers in response to exposure to UV-visible light, we studied the LCST properties of a homologous set of derivatives of poly(N-isopropylacrylamide). The chains contained a hydrophilic oligo(ethylene oxide) side group (at 20 mol%) and a light-responsive azobenzene introduced at a varying integration level (ranging from 0% to 24 mol%). Turbidity measurements showed that the LCST passes through a minimum value upon increasing the density of the hydrophobic azobenzene. At a low % of azobenzene, the LCST displayed the conventional decrease with increasing hydrophobicity of the polymers, from ca. 56 degrees C in the absence of azobenzene down to ca. 35 degrees C for the polymer containing 5 mol% azobenzene. Conversely, beyond 11 mol% azobenzene, the LCST increased upon increasing the hydrophobicity of the polymers, and a "reverse" response to exposure to light was observed. Namely the LCST decreased by up to 5 degrees C upon the photoconversion of azobenzene from the trans, apolar isomer, to the cis, polar, i.e. more water soluble isomer. This counterintuitive behavior was accompanied by the formation of clusters of chains, whose diameter (similar to 20 nm) was measured by dynamic light scattering. AFM and UV-visible spectra confirmed the presence of hydrophobic associates between the azobenzene groups. We ascribed the origin of the "reverse" responses to the presence of these clusters. We proposed accordingly a model of the partitioning of azobenzene between water and the hydrophobic core of the clusters, which allowed us to calculate the LCST as a function of polymer composition. Fitting experimental data to the model quantitatively indicated that self-assemblies scavenge the hydrophobic photochromes and in turn tune their contribution to the LCST.