Neural Networks最新文献

Learning from data streams that emerge from nonstationary environments has many real-world applications and poses various challenges. A key characteristic of such a task is the varying nature of the underlying data distributions over time (concept drifts). However, the most common type of data stream learning approach are ensemble approaches, which involve the training of multiple base learners. This can severely increase their computational cost, especially when the learners have to recover from concept drift, rendering them inadequate for applications with tight time and space constraints. In this work, we propose Online Weight Averaging (OWA) - a robust and fast online model compression method for nonstationary data streams based on stochastic weight averaging. It is the first online model compression for nonstationary data streams, which is capable of compressing an evolving ensemble of neural networks into a single model continuously over time. It combines several snapshots of a neural network over time by averaging its weights in specific time steps to find promising regions in the loss landscape with the ability to forget weights from outdated time steps when a concept drift occurs. In this way, at any point in time, a single neural network is maintained to represent a whole ensemble, leveraging the power of ensembles while being appropriate for applications with tight speed requirements. Our experiments show that this key advantage of our proposed method also translates into other advantages such as (1) significant savings in computational cost compared to state-of-the-art data stream ensemble methods while (2) delivering similar predictive performance.

Detecting anomalies in attributed networks has become a subject of interest in both academia and industry due to its wide spectrum of applications. Although most existing methods achieve desirable performance by the merit of various graph neural networks, the way they bundle node-affiliated multidimensional attributes into a whole for embedding calculation hinders their ability to model and analyze anomalies at the fine-grained feature level. To characterize anomalies from each feature dimension, we propose Eagle, a deep framework based on bipartitE grAph learninG for anomaLy dEtection. Specifically, we disentangle instances and attributes as two disjoint and independent node sets, then formulate the input attributed network as an intra-connected bipartite graph that involves two different relations: edges across two types of nodes described by attribute values, and links between nodes of the same type recorded in the network topology. By learning a self-supervised edge-level prediction task, named affinity inference, Eagle has good physical sense in explaining abnormal deviations from each attribute. Experiments corroborate the effectiveness of Eagle under transductive and inductive task settings. Moreover, case studies illustrate that Eagle is more user-friendly as it opens the door for humans to understand abnormalities from the perspective of different feature combinations.

Model-based diagnosis (MBD) is a critical problem in artificial intelligence. Recent advancements have made it possible to address this challenge using methods like deep learning. However, current approaches that use deep learning for MBD often struggle with accuracy and computation time due to the limited diagnostic information provided by a single observation. To address this challenge, we introduce two novel algorithms, Discret2DiMO (Discret2Di with Multiple Observations) and Discret2DiMO-DC (Discret2Di with Multiple Observations and Dictionary Cache), which enhance MBD by integrating multiple observations with deep learning techniques. Experimental evaluations on a simulated three-tank model demonstrate that Discret2DiMO significantly improves diagnostic accuracy, achieving up to a 685.06% increase and an average improvement of 59.18% over Discret2Di across all test cases. To address computational overhead, Discret2DiMO-DC additionally implements a caching mechanism that eliminates redundant computations during diagnosis. Remarkably, Discret2DiMO-DC achieves comparable accuracy while reducing computation time by an average of 95.74% compared to Discret2DiMO and 89.42% compared to Discret2Di, with computation times reduced by two orders of magnitude. These results indicate that our proposed algorithms significantly enhance diagnostic accuracy and efficiency in MBD compared with the state-of-the-art algorithm, highlighting the potential of integrating multiple observations with deep learning for more accurate and efficient diagnostics in complex systems.

Reconstruction-based methods achieve promising performance for visual anomaly detection (AD), relying on the underlying assumption that the anomalies cannot be accurately reconstructed. However, this assumption does not always hold, especially when suffering weak anomalous (a.k.a. normal-like) examples. More significantly, the existing methods primarily devote to obtaining the strong discriminative score functions, but neglecting the systematic investigation of the decision rule based on the proposed score function. Unlike previous work, this paper solves the AD issue starting from the decision rule within the statistical framework, providing a new insight for AD community. Specifically, we frame the AD task as a multiple hypothesis testing problem, Then, we propose a novel betting-like (BL) procedure with an embedding of strong anomaly constraint network (SACNet), called SAC-BL, to address this testing problem. In SAC-BL, BL procedure serves as the decision rule and SACNet is trained to capture the critical discriminative information from weak anomalies. Theoretically, our SAC-BL can control false discovery rate (FDR) at the prescribed level. Finally, we conduct extensive experiments to verify the superiority of SAC-BL over previous method.

Shadow removal remains a challenging visual task aimed at restoring the original brightness of shadow regions in images. Many existing methods overlook the implicit clues within non-shadow regions, leading to inconsistencies in the color, texture, and illumination of the reconstructed shadow-free images. To address these issues, we propose an efficient hybrid model of Transformer and Generative Adversarial Network (GAN), named ShadowGAN-Former, which utilizes information from non-shadow regions to assist in shadow removal. We introduce the Multi-Head Transposed Attention (MHTA) and Gated Feed-Forward Network (Gated FFN), designed to enhance focus on key features while reducing computational costs. Furthermore, we propose the Shadow Attention Reweight Module (SARM) to reweight the self-attention maps based on the correlation between shadow and non-shadow regions, thereby emphasizing the contextual relevance between them. Experimental results on the ISTD and SRD datasets show that our method outperforms popular and state-of-the-art shadow removal algorithms, with the SARM module improving PSNR by 5.42% and reducing RMSE by 14.76%.

Sequential recommendation models aim to predict the next item based on the sequence of items users interact with, ordered chronologically. However, these models face the challenge of data sparsity. Recent studies have explored cross-domain sequential recommendation, where users' interaction data across multiple source domains are leveraged to enhance recommendations in data-sparse target domains. Despite this, users' interests in the target and source domains may not align perfectly. Additionally, current research often neglects the collaboration between different transfer strategies across source domains, leading to suboptimal performance. To address these challenges, we propose a multi-agent reinforcement learning framework for cross-domain sequential recommendation (MARL4CDSR). Unlike traditional approaches that transfer knowledge from the entire source domain sequence, MARL4CDSR uses agents to select relevant items from source domain sequences for transfer. This approach optimizes the transfer process by coordinating agents' strategies within each source domain through a multi-agent reinforcement learning framework. Additionally, we introduce an information fusion module with a cross-attention mechanism to align the embedding representations of selected source domain items with target domain items. A reward function based on score differences for the next item optimizes the multi-agent system. We evaluate the method on three Amazon domains: Movies_and_TV, Toys_and_Games, and Books. Our proposed model MARL4CDSR outperforms all baselines on all metrics. Specifically, for the Movies&Books→Toys task, where the target domain interaction sequence is relatively sparse, MARL4CDSR improves NDCG@10 and HR@10 by 14.76% and 10.25%, respectively.

Unsupervised domain adaptation (UDA) aims to annotate unlabeled target domain samples using transferable knowledge learned from the labeled source domain. Optimal transport (OT) is a widely adopted probability metric in transfer learning for quantifying domain discrepancy. However, many existing OT-based UDA methods usually employ an entropic regularization term to solve the OT optimization problem, inevitably resulting in a biased estimation of domain discrepancy. Furthermore, to achieve precise alignment of class distributions, numerous UDA methods commonly employ deep features for guiding contrastive learning, overlooking the loss of discriminative information. Additionally, prior studies frequently use conditional entropy regularization term to cluster unlabeled target samples, which may guide the model toward optimizing in the wrong direction. To address the aforementioned issues, this paper proposes a new UDA framework called EHM, which employs a Dynamic Domain Alignment (DDA) strategy, a Reliable High-order Contrastive Alignment (RHCA) strategy, and a Trustworthy Hierarchical Clustering (THC) strategy. Specially, DDA leverages a dynamically adjusted Sinkhorn divergence to measure domain discrepancy, effectively eliminating the biased estimation issue. Our RHCA skillfully conducts contrastive learning in a high-order moment space, significantly enhancing the representation power of transferable features and reducing the domain discrepancy at the class-level. Moreover, THC integrates multi-view information to guide unlabeled samples towards achieving robust clustering. Extensive experiments on various benchmarks demonstrate the effectiveness of our EHM.

The attention towards clustering using anchor graph has grown due to its effectiveness and efficiency. As the most representative points in original data, anchors are also regarded as connecting the sample space to the label space. However, when there is noise in original data, the anchor-guided label spreading may fail. To alleviate this, we propose a Fast Co-clustering method via Anchor-guided Label Spreading (FCALS), in which the label of samples and anchors could be obtained simultaneously. Our method could not only maximize the intra-cluster similarity among anchors but also ensure that the relationship between anchors and original data is preserved. Besides, to avoid trivial solutions, the size constraint is introduced in our model, in which it is required that the samples within each cluster must not fall below a certain value. Furthermore, the lower limit exhibits insensitivity with a relatively broad range of possible values. Considering that the label matrix of original data could be fuzzy or discrete, the continuous and discrete models are proposed, which are named FCALS-C and FCALS-D respectively. Since labels of anchors can be directly obtained, the proposed methods are naturally applicable to out-of-sample problems. The superiority of the proposed methods is demonstrated through experimental results on both synthetic and real-world datasets.

In this paper, we derive diffusion equation models in the spectral domain to study the evolution of the training error of two-layer multiscale deep neural networks (MscaleDNN) (Cai and Xu, 2019; Liu et al., 2020), which is designed to reduce the spectral bias of fully connected deep neural networks in approximating oscillatory functions. The diffusion models are obtained from the spectral form of the error equation of the MscaleDNN, derived with a neural tangent kernel approach and gradient descent training and a sine activation function, assuming a vanishing learning rate and infinite network width and domain size. The involved diffusion coefficients are shown to have larger supports if more scales are used in the MscaleDNN, and thus, the proposed diffusion equation models in the frequency domain explain the MscaleDNN's spectral bias reduction capability. The diffusion model in the Fourier-spectral domain allows us to understand clearly the training error decay for different Fourier-frequencies. The numerical results of the diffusion models for a two-layer MscaleDNN training match the error evolution of the actual gradient descent training with a reasonably large network width, thus validating the effectiveness of the diffusion models. Meanwhile, the numerical results for MscaleDNN show error decay over a wide frequency range and confirm the advantage of using MscaleDNN to approximate functions with a wide range of frequencies.

We solve high-dimensional steady-state Fokker-Planck equations on the whole space by applying tensor neural networks. The tensor networks are a linear combination of tensor products of one-dimensional feedforward networks or a linear combination of several selected radial basis functions. The use of tensor feedforward networks allows us to efficiently exploit auto-differentiation (in physical variables) in major Python packages while using radial basis functions can fully avoid auto-differentiation, which is rather expensive in high dimensions. We then use the physics-informed neural networks and stochastic gradient descent methods to learn the tensor networks. One essential step is to determine a proper bounded domain or numerical support for the Fokker-Planck equation. To better train the tensor radial basis function networks, we impose some constraints on parameters, which lead to relatively high accuracy. We demonstrate numerically that the tensor neural networks in physics-informed machine learning are efficient for steady-state Fokker-Planck equations from two to ten dimensions.