Silicon nanostructures have a rich optical response thanks to Mie-type optical resonances, that can be designed on-demand via their geometry. It is possible to encode bits of information in a nanostructure’s geometry, and retrieve this information optically via the color observed in dark-field microscopy. Furthermore, asymmetric structures can profit from the illuminating light polarization to facilitate information readout. Our ultimate goal is to accurately reverse engineer experimentally feasible silicon nanostructures for information encoding, such that they implement a set of ideally distinguishable colors for robust optical readout. Deep learning is increasingly being used to solve inverse problems such as nano-photonic structure design. Neural networks for inverse design are mostly trained on simulated data, which is cheap to generate. But training neural networks on experimental data is a very interesting option, because it allows to include all experimental constraints into the model, which consequently learns to capture phenomena that may be hard to simulate. Here, in order to learn an accurate model for the full experimental measurement setup, we trained a neural network with experimental darkfield color data from several thousand nanostructures. Firstly, we built a forward network, taking as input the nanostructures’ shapes from fabricated samples and predicting the dark-field color for both X and Y polarizations. We then successfully built an inverse tandem network, capable of designing structures with desired color responses. In order to create distinguishable color responses, another deep neural network was trained on the task to map all experimental colors in a regularized color latent space. Sampling equidistant points from this latent space then yields the most distinguishable, yet experimentally feasible colors. The next future step will be to produce samples from the generated structures to test the network’s accuracy. We would like to test how many bits of information we can encode using the darkfield color as readout.