An overarching challenge in rheology is to develop constitutive models for complex fluids for which we lack accurate first principles theory. A further challenge is that most experiments probing dynamical structure and rheology do so only in very simple flow fields that are not characteristic of the complex deformation histories experienced by material in a processing application. A recently developed experimental methodology holds potential to overcome this challenge by incorporating a fluidic four-roll mill (FFoRM) into scanning small-angle X-ray scattering instrumentation (sSAXS) (Corona, P. T. et al. Sci. Rep. 8, 15559 (2018); Corona, P. T. et al. Phys. Rev. Mater 6, 045603 (2022)) to rapidly generate large data sets of scattering intensity for complex fluids along diverse Lagrangian flow histories. To exploit this uniquely rich FFoRM-sSAXS data, we propose a machine learning framework, Scattering-Informed Microstructure Prediction under Lagrangian Evolution (SIMPLE), which uses FFoRM-sSAXS data to learn an evolution equation for the scattering intensity and an associated tensorial differential constitutive equation for the stress. The framework incorporates material frame indifference and invariance to arbitrary rotations by data preprocessing. We use autoencoders to find an efficient reduced order model for the scattering intensity and neural network ordinary differential equations to predict the time evolution of the model coordinates. The framework is validated on a synthetic FFoRM-sSAXS data set for a dilute rigid rod suspension. The model accurately predicts microstructural evolution and rheology for flows that differ significantly from those used in training. SIMPLE is compatible with but does not require material-specific constraints or assumptions.