Immunohistochemistry annotations enhance AI identification of lymphocytes and neutrophils in digitized H&E slides from inflammatory bowel disease

Abstract

Background and Objective

Histologic assessment of the immune infiltrate in H&E slides is vital in diagnosing and managing inflammatory bowel diseases, but these assessments are subjective and time-consuming even for those with expertise. The development of deep learning models to aid in these assessments has been limited by the paucity of image data with reliably annotated immune cells available for training.

Methods

To address these challenges, we developed a pipeline that automates the neutrophil and lymphocyte labeling in ROIs from digital H&E slides. The data included ROIs extracted from 19 digitized H&E slides and the same slides restained with immunohistochemistry. Our pipeline first delineates each nucleus in H&E ROIs. Using the colorimetric features of the immunohistochemical stains (red: neutrophils, green: lymphocytes) in the immunohistochemistry ROIs, each cell was labeled as a neutrophil, a lymphocyte, or another cell. The labels were then transferred to the corresponding H&E ROIs by image registration, and the ROI registration accuracy was assessed by the median target registration error resulting in a labeled dataset. The newly formed dataset (NeuLy-IHC) comprising 519 ROIs with 235,256 labeled cells (74,339 lymphocytes, 16,326 neutrophils and 144,591 other cells) was used to train the HoVer-Net^(NeuLy) model. The performance of HoVer-Net^(NeuLy) measured by DICE coefficient (segmentation accuracy) and F1-scores (classification accuracy), was compared to those achieved by HoVer-Net^(MoNuSAC) and SMILE^(MoNuSAC) publicly available models trained on cancer-containing ROIs from the MoNuSAC dataset with manual cell labeling and pathologists’ annotations.

Results

The 1.0 μm median target registration error of ROIs observed was low demonstrating robust transferring of cellular labels from immunohistochemistry ROIs to H&E ROIs. In the test set comprising 76 NeuLy-IHC and 78 MoNuSAC ROIs, the HoVer-Net^(NeuLy) achieved a DICE coefficient of 0.861 and F1-sores of 0.827, 0.838, and 0.828, for neutrophils, lymphocytes, and other cells, respectively, outperforming the HoVer-Net^(MoNuSAC)'s and SMILE^(MoNuSAC)’s DICE coefficient and F1 scores for each cell category.

Conclusions

We attribute the improved performance of HoVer-Net^(NeuLy) to the larger number of immune cells in the NeuLy-IHC dataset (in total 5x more, including 21x more neutrophils) than in the MoNuSAC dataset. Despite being trained on data from inflammatory bowel disease specimens, our model maintained robust performance when tested on previously unseen data derived from cancer specimens. The NeuLy-IHC set provides opportunities for training accurate models to quantify the inflammatory infiltrate in digital histologic slides.