Journal of Parallel and Distributed Computing最新文献

Distributed learning with multiple GPUs has been widely adopted to accelerate the training process of large-scale deep neural networks. However, misconfiguration of the GPU clusters with various communication primitives and topologies could potentially diminish the gains in parallel computation and lead to significant degradation in training efficiency. Predicting the performance of distributed learning enables service providers to identify potential bottlenecks beforehand. In this work, we propose a Performance prediction framework over General Topologies, called PerfTop, for accurate estimation of per-iteration execution time. The main strategy is to integrate computation time prediction with an analytical model to map the nonlinearity in communication and fine-grained computation-communication patterns. This enables accurate prediction of a variety of neural network models over general topologies, such as tree, hierarchical, and exponential. Our extensive experiments show that PerfTop outperforms existing methods in estimating both computation and communication time, particularly for communication, surpassing the existing methods by over 45%. Meanwhile, it achieves an accuracy of above 85% in predicting the execution time over general topologies compared to simple topologies such as star and ring from the previous works.

Local Outlier Factor (LOF) is an unsupervised anomaly detection algorithm that finds anomalies by assessing the local density of a data point relative to its neighborhood. Anomaly detection is the process of finding anomalies in datasets. Anomalies in real-time datasets may indicate critical events like bank frauds, data compromise, network threats, etc. This paper deals with the implementation of the LOF algorithm in the HPCC Systems platform, which is an open-source distributed computing platform for big data analytics. Improved LOF is also proposed which efficiently detects anomalies in datasets rich in duplicates. The impact of varying hyperparameters on the performance of LOF is examined in HPCC Systems. This paper examines the performance of LOF with other algorithms like COF, LoOP, and kNN over several datasets in the HPCC Systems. Additionally, the efficacy of LOF is evaluated across big-data frameworks such as Spark, Hadoop, and HPCC Systems, by comparing their runtime performances.

Influence maximization (IM) is a combinatorial problem of identifying a subset of seed nodes in a network (graph), which when activated, provide a maximal spread of influence in the network for a given diffusion model and a budget for seed set size. IM has numerous applications such as viral marketing, epidemic control, sensor placement and other network-related tasks. However, its practical uses are limited due to the computational complexity of current algorithms. Recently, deep reinforcement learning has been leveraged to solve IM in order to ease the computational burden. However, there are serious limitations in current approaches, including narrow IM formulation that only consider influence via spread and ignore self activation, low scalability to large graphs, and lack of generalizability across graph families leading to a large running time for every test network. In this work, we address these limitations through a unique approach that involves: (1) Formulating a generic IM problem as a Markov decision process that handles both intrinsic and influence activations; (2) incorporating generalizability via meta-learning across graph families. There are previous works that combine deep reinforcement learning with graph neural network but this work solves a more realistic IM problem and incorporates generalizability across graphs via meta reinforcement learning. Extensive experiments are carried out in various standard networks to validate performance of the proposed Graph Meta Reinforcement learning (GraMeR) framework. The results indicate that GraMeR is multiple orders faster and generic than conventional approaches when applied on small to medium scale graphs.

We design, develop and analyze parallel variants of a state-of-the-art graybox optimization algorithm, namely Drils (Deterministic Recombination and Iterated Local Search), for attacking large-scale pseudo-boolean optimization problems on top of the large-scale computing facilities offered by the supercomputer Fugaku. We first adopt a Master/Worker design coupled with a fully distributed Island-based model, ending up with a number of hybrid OpenMP/MPI implementations of high-level parallel Drils versions. We show that such a design, although effective, can be substantially improved by enabling a more focused iteration-level cooperation mechanism between the core graybox components of the original serial Drils algorithm. Extensive experiments are conducted in order to provide a systematic analysis of the impact of the designed parallel algorithms on search behavior, and their ability to compute high-quality solutions using increasing number of CPU-cores. Results using up to 1024×12-cores NUMA nodes, and NK-landscapes with up to $10,000$ binary variables are reported, providing evidence on the relative strength of the designed hybrid cooperative graybox parallel search.

Cache memories play a significant role in the performance, area, and energy consumption of modern processors, and this impact is expected to grow as on-die memories become larger. While caches are highly effective for cache-friendly access patterns, they introduce unnecessary delays and energy wastage when they fail to serve the required data. Hence, cache bypassing techniques have been proposed to optimize the latency of cache-unfriendly memory accesses. In this scenario, we discuss HBPB, a history-based preemptive bypassing technique that accelerates cache-unfriendly access through the reduced latency of bypassing the caches. By extensively evaluating different real-world applications and hardware cache configurations, we show that HBPB yields energy reductions of up to 75% and performance improvements of up to 50% compared to a version that does not apply cache bypassing. More importantly, we demonstrate that HBPB does not affect the performance of applications with cache-friendly access patterns.

Federated learning (FL) as a novel paradigm in Artificial Intelligence (AI), ensures enhanced privacy by eliminating data centralization and brings learning directly to the edge of the user's device. Nevertheless, new privacy issues have been raised particularly during training and the exchange of parameters between servers and clients. While several privacy-preserving FL solutions have been developed to mitigate potential breaches in FL architectures, their integration poses its own set of challenges. Incorporating these privacy-preserving mechanisms into FL at the edge computing level can increase both communication and computational overheads, which may, in turn, compromise data utility and learning performance metrics. This paper provides a systematic literature review on essential methods and metrics to support the most appropriate trade-offs between FL privacy and other performance-related application requirements such as accuracy, loss, convergence time, utility, communication, and computation overhead. We aim to provide an extensive overview of recent privacy-preserving mechanisms in FL used across various applications, placing a particular focus on quantitative privacy assessment approaches in FL and the necessity of achieving a balance between privacy and the other requirements of real-world FL applications. This review collects, classifies, and discusses relevant papers in a structured manner, emphasizing challenges, open issues, and promising research directions.

Cloud computing delivers a desirable environment for users to run their different kinds of applications in a cloud. Numerous of these applications (tasks), such as bioinformatics, astronomy, biodiversity, and image analysis, are deadline-sensitive. Such tasks must be properly allocated to virtual machines (VMs) to avoid deadline violations, and they should reduce their execution time and cost. Due to the contradictory environment, minimizing the application task's completion time and execution cost is extremely difficult. Thus, we propose a Cost-aware Quantum-inspired Genetic Algorithm (CQGA) to minimize the execution time and cost by meeting the deadline constraints. CQGA is motivated by quantum computing and genetic algorithm. It combines quantum operators (measure, interference, and rotation) with genetic operators (selection, crossover, and mutation). Quantum operators are used for better population diversity, quick convergence, time-saving, and robustness. Genetic operators help to produce new individuals, have good fitness values for individuals, and play a significant role in preserving the evolution quality of the population. In addition, CQGA used a quantum bit as a probabilistic representation because it has higher population diversity attributes than other representations. The simulation outcome exhibits that the proposed algorithm can obtain outstanding convergence performance and reduced maximum cost than benchmark algorithms.

Given the distinctive characteristics of flash-based solid-state drives (SSDs), such as out-of-place update scheme, as compared to traditional block storage devices, a flash translation layer (FTL) has been introduced to hide these features. In the FTL, there is an address translation module that implements the conversion from logical addresses to physical addresses. However, existing address mapping algorithms currently fail to fully exploit the mapping information generated by large I/O requests. First, based on this observation, we propose a novel continuity compressed page-level flash address mapping method (CCFTL). This method effectively compresses the mapping relationship between consecutive logical addresses and physical addresses, enabling the storage of more mapping information within the same mapping cache size. Next, we introduce two-level LRU linked list to mitigate the issue of compressed mapping entry splitting that arises from handling write requests. Finally, our experiments show that CCFTL reduced average response times by 52.67%, 16.81%, and 12.71% compared to DFTL, TPFTL, and MFTL, respectively. As the mapping cache size decreases from 2 MB to 1 MB, then further decreases to 256 KB, 128 KB, and eventually down to 64 KB, CCFTL experiences an average decline ratio of less than 3% in average response time, while the other three algorithms show an average decline ratio of 9.51%.

Distributedly training models across diverse clients with heterogeneous data samples can significantly impact the convergence of federated learning. Various novel federated learning methods address these challenges but often require significant communication resources and local computational capacity, leading to reduced global inference accuracy in scenarios with imbalanced label data distribution and quantity skew. To tackle these challenges, we propose FedVGL, a Federated Variational Generative Learning method that directly trains a local generative model to learn the distribution of local features and improve global target model inference accuracy during aggregation, particularly under conditions of severe data heterogeneity. FedVGL facilitates distributed learning by sharing generators and latent vectors with the global server, aiding in global target model training from mapping local data distribution to the variational latent space for feature reconstruction. Additionally, FedVGL implements anonymization and encryption techniques to bolster privacy during generative model transmission and aggregation. In comparison to vanilla federated learning, FedVGL minimizes communication overhead, demonstrating superior accuracy even with minimal communication rounds. It effectively mitigates model drift in scenarios with heterogeneous data, delivering improved target model training outcomes. Empirical results establish FedVGL's superiority over baseline federated learning methods under severe label imbalance and data skew condition. In a Label-based Dirichlet Distribution setting with α=0.01 and 10 clients using the MNIST dataset, FedVGL achieved an exceptional accuracy over 97% with the VGG-9 target model.

Triple modular redundancy (TMR) fault tolerance mechanism can provide almost perfect fault-masking, which has the great potential to enhance the reliability of real-time systems. However, multiple copies of a task are executed concurrently, which will lead to a sharp increase in system energy consumption. In this work, the problem of parallel applications using TMR on heterogeneous multi-core platforms to minimize energy consumption is studied. First, the heterogeneous earliest finish time algorithm is improved, and then according to the given application's deadline constraints and reliability requirements, an algorithm to extend the execution time of the copies is designed. Secondly, based on the properties of TMR, an algorithm for minimizing the execution overhead of the third copy (MEOTC) is designed. Finally, considering the actual situation of task execution, an online energy management (OEM) method is proposed. The proposed algorithms were compared with the state-of-the-art AFTSA algorithm, and the results show significant differences in energy consumption. Specifically, for light fault detection, the energy consumption of the MEOTC and OEM algorithms was found to be 80% and 72% respectively, compared with AFTSA. For heavy fault detection, the energy consumption of MEOTC and OEM was measured at 61% and 55% respectively, compared with AFTSA.