Journal of Parallel and Distributed Computing最新文献

The rapid development of artificial intelligence and networking technologies has catalyzed the popularity of intelligent services based on deep learning in recent years, which in turn fosters the advancement of Web of Things (WoT). Big social data (BSD) plays an important role during the processing of intelligent services in WoT. However, intelligent BSD services are computationally intensive and require ultra-low latency. End or edge devices with limited computing power cannot realize the extremely low response latency of those services. Distributed inference of deep neural networks (DNNs) on various devices is considered a feasible solution by allocating the computing load of a DNN to several devices. In this work, an efficient model parallelism method that couples convolution layer (Conv) split with resource allocation is proposed. First, given a random computing resource allocation strategy, the Conv split decision is made through a mathematical analysis method to realize the parallel inference of convolutional neural networks (CNNs). Next, Deep Reinforcement Learning is used to get the optimal computing resource allocation strategy to maximize the resource utilization rate and minimize the CNN inference latency. Finally, simulation results show that our approach performs better than the baselines and is applicable for BSD services in WoT with a high workload.

Big data frameworks are widely deployed in supercomputers for analyzing large-scale datasets. Topological data processing is an emerging approach that focuses on analyzing the topological structures in high-dimensional scientific data. However, incorporating topological data processing into current big data frameworks presents three main challenges: (1) The frequent data exchange poses challenges to the traditional coarse-grained parallelism. (2) The spatial topology makes parallel programming harder using oversimplified MapReduce APIs. (3) The massive intermediate data and NUMA architecture hinder resource utilization and scalability on novel supercomputers and many-core processors.

In this paper, we present Topo, a generic distributed framework that enhances topological data processing on many-core supercomputers. Topo relies on three concepts. (1) It employs fine-grained parallelism, with awareness of topological structures in datasets, to support interactions among collaborative workers before each shuffle phase. (2) It provides intuitive APIs for topological data operations. (3) It implements efficient collective I/O and NUMA-aware dynamic task scheduling to improve multi-threading and load balancing. We evaluate Topo's performance on the Tianhe-3 supercomputer, which utilizes state-of-the-art ARM many-core processors. Experimental results of execution time show that compared to popular frameworks, Topo achieves an average speedup of 5.3× and 6.3×, with a maximum speedup of 8.4× and 20×, on HPC workloads and big data benchmarks, respectively. Topo further reduces total execution time on processing skewed datasets by 41%.

The folded hypercube is one of the hypercube variants and is of great significance in the study of interconnection networks. In a folded hypercube, information can be broadcast using efficient distributed algorithms. In the context of parallel computing, folded hypercube has been studied as a possible network topology as an alternative to the hypercube. The routing and wavelength assignment (RWA) problem is significant, since it improves the performance of wavelength-routed all-optical networks constructed using wavelength division multiplexing approach. Given the physical network topology, the aim of the RWA problem is to establish routes for the connection requests and assign the fewest possible wavelengths in accordance with the wavelength continuity and distinct wavelength constraints. This paper discusses the RWA problem in a linear array for the folded hypercube communication pattern by using the congestion technique.

Recent hardware-aware matrix-free algorithms for higher-order finite-element (FE) discretized matrix-vector multiplications reduce floating point operations and data access costs compared to traditional sparse matrix approaches. In this work, we address a critical gap in existing matrix-free implementations which are not well suited for the action of FE discretized matrices on very large number of vectors. In particular, we propose efficient matrix-free algorithms for evaluating FE discretized matrix-multivector products on both multi-node CPU and GPU architectures. To this end, we employ batched evaluation strategies, with the batchsize tailored to underlying hardware architectures, leading to better data locality and enabling further parallelization. On CPUs, we utilize even-odd decomposition, SIMD vectorization, and overlapping computation and communication strategies. On GPUs, we develop strategies to overlap compute with data movement for achieving efficient pipelining and reduced data accesses through the use of GPU-shared memory, constant memory and kernel fusion. Our implementation outperforms the baselines for Helmholtz operator action on 1024 vectors, achieving up to 1.4x improvement on one CPU node and up to 2.8x on one GPU node, while reaching up to 4.4x and 1.5x improvement on multiple nodes for CPUs (3072 cores) and GPUs (24 GPUs), respectively. We further benchmark the performance of the proposed implementation for solving a model eigenvalue problem for 1024 smallest eigenvalue-eigenvector pairs by employing the Chebyshev Filtered Subspace Iteration method, achieving up to 1.5x improvement on one CPU node and up to 2.2x on one GPU node while reaching up to 3.0x and 1.4x improvement on multi-node CPUs (3072 cores) and GPUs (24 GPUs), respectively.

Distributed learning with multiple GPUs has been widely adopted to accelerate the training process of large-scale deep neural networks. However, misconfiguration of the GPU clusters with various communication primitives and topologies could potentially diminish the gains in parallel computation and lead to significant degradation in training efficiency. Predicting the performance of distributed learning enables service providers to identify potential bottlenecks beforehand. In this work, we propose a Performance prediction framework over General Topologies, called PerfTop, for accurate estimation of per-iteration execution time. The main strategy is to integrate computation time prediction with an analytical model to map the nonlinearity in communication and fine-grained computation-communication patterns. This enables accurate prediction of a variety of neural network models over general topologies, such as tree, hierarchical, and exponential. Our extensive experiments show that PerfTop outperforms existing methods in estimating both computation and communication time, particularly for communication, surpassing the existing methods by over 45%. Meanwhile, it achieves an accuracy of above 85% in predicting the execution time over general topologies compared to simple topologies such as star and ring from the previous works.

Local Outlier Factor (LOF) is an unsupervised anomaly detection algorithm that finds anomalies by assessing the local density of a data point relative to its neighborhood. Anomaly detection is the process of finding anomalies in datasets. Anomalies in real-time datasets may indicate critical events like bank frauds, data compromise, network threats, etc. This paper deals with the implementation of the LOF algorithm in the HPCC Systems platform, which is an open-source distributed computing platform for big data analytics. Improved LOF is also proposed which efficiently detects anomalies in datasets rich in duplicates. The impact of varying hyperparameters on the performance of LOF is examined in HPCC Systems. This paper examines the performance of LOF with other algorithms like COF, LoOP, and kNN over several datasets in the HPCC Systems. Additionally, the efficacy of LOF is evaluated across big-data frameworks such as Spark, Hadoop, and HPCC Systems, by comparing their runtime performances.

Influence maximization (IM) is a combinatorial problem of identifying a subset of seed nodes in a network (graph), which when activated, provide a maximal spread of influence in the network for a given diffusion model and a budget for seed set size. IM has numerous applications such as viral marketing, epidemic control, sensor placement and other network-related tasks. However, its practical uses are limited due to the computational complexity of current algorithms. Recently, deep reinforcement learning has been leveraged to solve IM in order to ease the computational burden. However, there are serious limitations in current approaches, including narrow IM formulation that only consider influence via spread and ignore self activation, low scalability to large graphs, and lack of generalizability across graph families leading to a large running time for every test network. In this work, we address these limitations through a unique approach that involves: (1) Formulating a generic IM problem as a Markov decision process that handles both intrinsic and influence activations; (2) incorporating generalizability via meta-learning across graph families. There are previous works that combine deep reinforcement learning with graph neural network but this work solves a more realistic IM problem and incorporates generalizability across graphs via meta reinforcement learning. Extensive experiments are carried out in various standard networks to validate performance of the proposed Graph Meta Reinforcement learning (GraMeR) framework. The results indicate that GraMeR is multiple orders faster and generic than conventional approaches when applied on small to medium scale graphs.

We design, develop and analyze parallel variants of a state-of-the-art graybox optimization algorithm, namely Drils (Deterministic Recombination and Iterated Local Search), for attacking large-scale pseudo-boolean optimization problems on top of the large-scale computing facilities offered by the supercomputer Fugaku. We first adopt a Master/Worker design coupled with a fully distributed Island-based model, ending up with a number of hybrid OpenMP/MPI implementations of high-level parallel Drils versions. We show that such a design, although effective, can be substantially improved by enabling a more focused iteration-level cooperation mechanism between the core graybox components of the original serial Drils algorithm. Extensive experiments are conducted in order to provide a systematic analysis of the impact of the designed parallel algorithms on search behavior, and their ability to compute high-quality solutions using increasing number of CPU-cores. Results using up to 1024×12-cores NUMA nodes, and NK-landscapes with up to $10,000$ binary variables are reported, providing evidence on the relative strength of the designed hybrid cooperative graybox parallel search.

Cache memories play a significant role in the performance, area, and energy consumption of modern processors, and this impact is expected to grow as on-die memories become larger. While caches are highly effective for cache-friendly access patterns, they introduce unnecessary delays and energy wastage when they fail to serve the required data. Hence, cache bypassing techniques have been proposed to optimize the latency of cache-unfriendly memory accesses. In this scenario, we discuss HBPB, a history-based preemptive bypassing technique that accelerates cache-unfriendly access through the reduced latency of bypassing the caches. By extensively evaluating different real-world applications and hardware cache configurations, we show that HBPB yields energy reductions of up to 75% and performance improvements of up to 50% compared to a version that does not apply cache bypassing. More importantly, we demonstrate that HBPB does not affect the performance of applications with cache-friendly access patterns.