Growing evidence suggests that organisms with narrow niche requirements are particularly disadvantaged in small habitat patches, typical of fragmented landscapes. However, the mechanisms behind this relationship remain unclear. Dietary specialists may be particularly constrained by the availability of their food resources as habitat area shrinks. For herbivorous insects, host plants may be filtered out of small habitat fragments by neutral sampling processes and deterministic plant community shifts due to altered microclimates, edge effects and browsing by ungulates. We examined the relationship between forest fragment area and the abundance of dietary-specialist and dietary-generalist larval Lepidoptera (caterpillars) and their host plants in the northeastern USA. We surveyed caterpillars and their host plants over 3 years in equal-sized plots within 32 forest fragments varying in area between 3 and 1014 ha. We tested whether the abundances and species richness of dietary specialists increased more than those of dietary generalists with increasing fragment area and, if so, whether the difference could be explained by reduced host plant availability or increased browsing by white-tailed deer (Odocoileus virginianus). The overall abundance of dietary specialists was positively related to fragment area; the relationship was substantially weaker for dietary generalists. There was notable variation among species within diet breadth groups, however. There was no effect of fragment area on the diversity of dietary-specialist or dietary-generalist caterpillars. Deer activity was not related to the abundances of either dietary-generalist or dietary-specialist caterpillars. Plant community composition was strongly associated with fragment area. Larger fragments were more likely to include host plants for both dietary-specialist and dietary-generalist caterpillars. Deer activity was correlated with decreased host plant availability for both groups, with a slightly stronger impact on host plants of dietary specialists. Although dietary specialists were more likely to lack host plants in fragments, the relationship between fragment area and host availability did not depend on caterpillar diet breadth. This study provides further evidence that decreasing patch area disproportionately impacts specialist consumers. Because this relationship was derived from equal-sized plots, it is robust to some criticisms levelled at fragmentation research. The mechanisms for specialist consumer declines, however, remain elusive.