{"title":"持续学习的潜谱正则化","authors":"Emanuele Frascaroli , Riccardo Benaglia , Matteo Boschini , Luca Moschella , Cosimo Fiorini , Emanuele Rodolà , Simone Calderara","doi":"10.1016/j.patrec.2024.06.020","DOIUrl":null,"url":null,"abstract":"<div><p>While biological intelligence grows organically as new knowledge is gathered throughout life, Artificial Neural Networks forget catastrophically whenever they face a changing training data distribution. Rehearsal-based Continual Learning (CL) approaches have been established as a versatile and reliable solution to overcome this limitation; however, sudden input disruptions and memory constraints are known to alter the consistency of their predictions. We study this phenomenon by investigating the geometric characteristics of the learner’s latent space and find that replayed data points of different classes increasingly mix up, interfering with classification. Hence, we propose a geometric regularizer that enforces weak requirements on the Laplacian spectrum of the latent space, promoting a partitioning behavior. Our proposal, called Continual Spectral Regularizer for Incremental Learning (CaSpeR-IL), can be easily combined with any rehearsal-based CL approach and improves the performance of SOTA methods on standard benchmarks.</p></div>","PeriodicalId":54638,"journal":{"name":"Pattern Recognition Letters","volume":null,"pages":null},"PeriodicalIF":3.9000,"publicationDate":"2024-06-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0167865524001909/pdfft?md5=843aa8ad438387d8ae33f617b9e9e4d3&pid=1-s2.0-S0167865524001909-main.pdf","citationCount":"0","resultStr":"{\"title\":\"Latent spectral regularization for continual learning\",\"authors\":\"Emanuele Frascaroli , Riccardo Benaglia , Matteo Boschini , Luca Moschella , Cosimo Fiorini , Emanuele Rodolà , Simone Calderara\",\"doi\":\"10.1016/j.patrec.2024.06.020\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><p>While biological intelligence grows organically as new knowledge is gathered throughout life, Artificial Neural Networks forget catastrophically whenever they face a changing training data distribution. Rehearsal-based Continual Learning (CL) approaches have been established as a versatile and reliable solution to overcome this limitation; however, sudden input disruptions and memory constraints are known to alter the consistency of their predictions. We study this phenomenon by investigating the geometric characteristics of the learner’s latent space and find that replayed data points of different classes increasingly mix up, interfering with classification. Hence, we propose a geometric regularizer that enforces weak requirements on the Laplacian spectrum of the latent space, promoting a partitioning behavior. Our proposal, called Continual Spectral Regularizer for Incremental Learning (CaSpeR-IL), can be easily combined with any rehearsal-based CL approach and improves the performance of SOTA methods on standard benchmarks.</p></div>\",\"PeriodicalId\":54638,\"journal\":{\"name\":\"Pattern Recognition Letters\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":3.9000,\"publicationDate\":\"2024-06-22\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.sciencedirect.com/science/article/pii/S0167865524001909/pdfft?md5=843aa8ad438387d8ae33f617b9e9e4d3&pid=1-s2.0-S0167865524001909-main.pdf\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Pattern Recognition Letters\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S0167865524001909\",\"RegionNum\":3,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Pattern Recognition Letters","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0167865524001909","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0
摘要
生物智能会随着一生中新知识的收集而有机增长,而人工神经网络则会在面对不断变化的训练数据分布时发生灾难性遗忘。基于排练的持续学习(CL)方法已被确立为克服这一局限性的通用而可靠的解决方案;然而,众所周知,突然的输入中断和记忆限制会改变其预测的一致性。我们通过研究学习者潜在空间的几何特征来研究这一现象,发现不同类别的重放数据点越来越多地混合在一起,从而干扰了分类。因此,我们提出了一种几何正则器,对潜在空间的拉普拉斯频谱执行弱要求,促进分区行为。我们的建议被称为增量学习的连续谱正则化器(CaSpeR-IL),可以很容易地与任何基于演练的 CL 方法相结合,并提高 SOTA 方法在标准基准上的性能。
Latent spectral regularization for continual learning
While biological intelligence grows organically as new knowledge is gathered throughout life, Artificial Neural Networks forget catastrophically whenever they face a changing training data distribution. Rehearsal-based Continual Learning (CL) approaches have been established as a versatile and reliable solution to overcome this limitation; however, sudden input disruptions and memory constraints are known to alter the consistency of their predictions. We study this phenomenon by investigating the geometric characteristics of the learner’s latent space and find that replayed data points of different classes increasingly mix up, interfering with classification. Hence, we propose a geometric regularizer that enforces weak requirements on the Laplacian spectrum of the latent space, promoting a partitioning behavior. Our proposal, called Continual Spectral Regularizer for Incremental Learning (CaSpeR-IL), can be easily combined with any rehearsal-based CL approach and improves the performance of SOTA methods on standard benchmarks.
期刊介绍:
Pattern Recognition Letters aims at rapid publication of concise articles of a broad interest in pattern recognition.
Subject areas include all the current fields of interest represented by the Technical Committees of the International Association of Pattern Recognition, and other developing themes involving learning and recognition.