Self-imposed short sleep durations are increasingly commonplace in society, and have considerable health and performance implications for individuals. Reduced sleep duration over multiple nights has similar behavioural effects to those observed following acute total sleep deprivation, suggesting that lack of sleep affects brain function cumulatively. A link between habitual sleep patterns and functional connectivity has previously been observed, and the effect of sleep duration on the brain's intrinsic functional architecture may provide a link between sleep status and cognition. However, it is currently not known whether differences in habitual sleep patterns across individuals are related to changes in the brain's white matter, which underlies structural connectivity. In the present study we use diffusion–weighted imaging and a group comparison application of tract based spatial statistics (TBSS) to investigate changes to fractional anisotropy (FA) and mean diffusivity (MD) in relation to sleep duration and quality, hypothesising that white matter metrics would be positively associated with sleep duration and quality. Diffusion weighted imaging data was acquired from a final cohort of 33 (23–29 years, 10 female, mean 25.4 years) participants. Sleep patterns were assessed for a 14 day period using wrist actigraphs and sleep diaries, and subjective sleep quality with the Pittsburgh Sleep Quality Index (PSQI). Median splits based on total sleep time and PSQI were used to create groups of shorter/longer and poorer/better sleepers, whose imaging data was compared using TBSS followed by post-hoc correlation analysis in regions identified as significantly different between the groups. There were significant positive correlations between sleep duration and FA in the left orbito-frontal region and the right superior corona radiata, and significant negative correlations between sleep duration and MD in right orbito-frontal white matter and the right inferior longitudinal fasciculus. Improved sleep quality was positively correlated with FA in left caudate nucleus, white matter tracts to the left orbito-frontal region, the left anterior cingulum bundle and the white matter tracts associated with the right operculum and insula, and negatively correlated with MD in left orbito-frontal white matter and the left anterior cingulum bundle. Our findings suggest that reduced cumulative total sleep time (cTST) and poorer subjective sleep quality are associated with subtle white matter micro-architectural changes. The regions we identified as being related to habitual sleep patterns were restricted to the frontal and temporal lobes, and the functions they support are consistent with those which have previously been demonstrated as being affected by short sleep durations (e.g., attention, cognitive control, memory). Examining how inter-individual differences in brain structure are related to habitual sleep patterns could help to she