Performance Evaluation最新文献

Internet service providers (ISPs) have a variety of quality attributes that determine their attractiveness for data transmission, ranging from quality-of-service metrics such as jitter to security properties such as the presence of DDoS defense systems. ISPs should optimize these attributes in line with their profit objective, i.e., maximize revenue from attracted traffic while minimizing attribute-related cost, all in the context of alternative offers by competing ISPs. However, this attribute optimization is difficult not least because many aspects of ISP competition are barely understood on a systematic level, e.g., the multi-dimensional and cost-driving nature of path quality, and the distributed decision making of ISPs on the same path.

In this paper, we improve this understanding by analyzing how ISP competition affects path quality and ISP profits. To that end, we develop a game-theoretic model in which ISPs (i) affect path quality via multiple attributes that entail costs, (ii) are on paths together with other selfish ISPs, and (iii) are in competition with alternative paths when attracting traffic. The model enables an extensive theoretical analysis, surprisingly showing that competition can have both positive and negative effects on path quality and ISP profits, depending on the network topology and the cost structure of ISPs. However, a large-scale simulation, which draws on real-world data to instantiate the model, shows that the positive effects will likely prevail in practice: If the number of selectable paths towards any destination increases from 1 to 5, the prevalence of quality attributes increases by at least 50%, while 75% of ISPs improve their profit.

Dispatching policies such as join the shortest queue (JSQ), join the queue with smallest workload (JSW), and their power of two variants are used in load balancing systems where the instantaneous queue length or workload information at all queues or a subset of them can be queried. In situations where the dispatcher has an associated memory, one can minimize this query overhead by maintaining a list of idle servers to which jobs can be dispatched. Recent alternative approaches that do not require querying such information include the cancel-on-start and cancel-on-complete replication policies. The downside of such policies however is that the servers must communicate either the start or the completion time instant of each service to the dispatcher and must allow the coordinated and instantaneous cancellation of all redundant replicas. In practice, the requirements of query messaging, memory, and replica cancellation pose challenges in their implementation and their advantages are not clear. In this work, we consider load-balancing policies that do not need to query load information, do not need memory, and do not need to cancel replicas. Our policies allow the dispatcher to append a timer to each job or its replica. A job or a replica is discarded if its timer expires before it starts receiving service. We analyze several variants of this policy which are novel and simple to implement. We numerically observe that the variants of the proposed policy outperform popular feedback-based policies for low arrival rates, despite no feedback from servers to the dispatcher.

Multiserver-job (MSJ) systems, where jobs need to run concurrently across many servers, are increasingly common in practice. The default service ordering in many settings is First-Come First-Served (FCFS) service. Virtually all theoretical work on MSJ FCFS models focuses on characterizing the stability region, with almost nothing known about mean response time.

We derive the first explicit characterization of mean response time in the MSJ FCFS system. Our formula characterizes mean response time up to an additive constant, which becomes negligible as arrival rate approaches throughput, and allows for general phase-type job durations.

We derive our result by utilizing two key techniques: REduction to Saturated for Expected Time (RESET) and MArkovian Relative Completions (MARC).

Using our novel RESET technique, we reduce the problem of characterizing mean response time in the MSJ FCFS system to an M/M/1 with Markovian service rate (MMSR). The Markov chain controlling the service rate is based on the saturated system, a simpler closed system which is far more analytically tractable.

Unfortunately, the MMSR has no explicit characterization of mean response time. We therefore use our novel MARC technique to give the first explicit characterization of mean response time in the MMSR, again up to constant additive error. We specifically introduce the concept of “relative completions,” which is the cornerstone of our MARC technique.

Resource management in microservices is challenging due to the uncertain latency–resource relationship, dynamic environment, and strict Service-Level Agreement (SLA) guarantees. This paper presents a Pessimistic and Optimistic Bayesian Optimization framework, named POBO, for safe and optimal resource configuration for microservice applications. POBO leverages Bayesian learning to estimate the uncertain latency–resource functions and combines primal–dual and penalty-based optimization to maximize resource efficiency while guaranteeing strict SLAs. We prove that POBO can achieve sublinear regret and SLA violation against the optimal resource configuration in hindsight. We have implemented a prototype of POBO and conducted extensive experiments on a real-world microservice application. Our results show that POBO can find the safe and optimal configuration efficiently, outperforming Kubernetes’ built-in auto-scaling module and the state-of-the-art algorithms.

In this paper, we develop a synthetic workload model for the Zoom network application based on empirical Zoom traffic measurements from a campus network. We then use this model in a simulation study of Zoom network traffic at the campus scale. The simulation results show that hybrid learning places a substantial load on the campus network. Additional simulation experiments investigate the potential benefits of locally-hosted Zoom infrastructure, improved load balancing strategies for Zoom servers, and multicast delivery for Zoom network traffic. The simulation results show that the multicast approach offers the greatest potential benefit for improving Zoom performance on our campus network.

We consider a single source–destination pair, where information updates (in short, updates) arrive at the source at arbitrary time instants. For each update, its size, i.e. the service time required for complete transmission to the destination, is also arbitrary. At any time, the source may choose which update to transmit, while incurring transmission cost that is proportional to the duration of transmission. We consider the age of information (AoI) metric that quantifies the staleness of the update (information) at the destination. At any time, AoI is equal to the difference between the current time, and the arrival time of the latest update (at the source) that has been completely transmitted (to the destination). The goal is to find a causal (i.e. online) scheduling policy that minimizes the sum of the AoI and the transmission cost, where the possible decisions at any time are (i) whether to preempt the update under transmission upon arrival of a new update, and (ii) if no update is under transmission, then choose which update to transmit (among the available updates). In this paper, we propose a causal policy called SRPT${}^{+}$ that at each time, (i) preempts the update under transmission if a new update arrives with a smaller size (compared to the remaining size of the update under transmission), and (ii) if no update is under transmission, then from the set of available updates with size less than a threshold (which is a function of the transmission cost and the current AoI), begins to transmit the update for which the ratio of the reduction in AoI upon complete transmission (if not preempted in future) and the remaining size, is maximum. We characterize the performance of SRPT${}^{+}$ using a metric called the competitive ratio, i.e. the ratio of the cost of causal policy and the cost of an optimal offline policy (that knows the entire input in advance), maximized over all possible inputs. We show that the competitive ratio of SRPT${}^{+}$ is at most 5. In the special case when there is no transmission cost, we further show that the competitive ratio of SRPT${}^{+}$ is at most 3.

We consider a multiserver queue where jobs request for a varying number of servers for a random service time. The requested number of servers is assigned to each job following a First-In First-Out (FIFO) order. When the number of free servers is not sufficient to accommodate the next job in line, that job and any subsequent jobs in the queue are forced to wait. As a result, not all available servers are allocated to jobs if the next job requires more servers than are currently free. This queuing system is often called a Multiserver Job Queuing Model (MJQM).

In this paper, we study the behavior of a MJQM under saturation, i.e., when the waiting line always contains jobs to be served. We categorize jobs into two classes: the first class consists of jobs that only require one server, while the second class includes jobs that require a larger number of servers. We obtain the system utilization and the throughput of the two job classes for the case in which the number of servers requested by jobs in the second class is equal to the number of available servers, using a simple approach that allows for a general distribution of the service time of jobs in the second class. Hence, we derive the stability condition of the non-saturated MJQM under these assumptions. Additionally, we develop an approximate analysis for the case in which the jobs of the second class require a fraction of the available servers.

Based on analytical and numerical results, we highlight interesting system properties and insights.

We construct a practical and real-time probabilistic framework for fine target tracking. In our scenario, a Bluetooth Low-Energy (BLE) device navigating in the environment publishes BLE packets that are captured by stationary BLE sensors. The aim is to accurately estimate the live position of the BLE device emitting these packets. The framework is built upon a hidden Markov model (HMM), the components of which are determined with a combination of heuristic and data-driven approaches. In the data-driven part, we rely on the fingerprints formed priorly by extracting received signal strength indicators (RSSI) from the packets. These data are then transformed into probabilistic radio-frequency maps that are used for measuring the likelihood between an RSSI data and a position. The heuristic part involves the movement of the tracked object. Having no access to any inertial information of the object, this movement is modeled with Gaussian densities with variable model parameters that are to be determined heuristically. The practicality of the framework comes from the associated small parameter set used to discretize the components of the HMM. By tuning these parameters, such as the grid size of the area, the mask size and the covariance of the Gaussian; a probabilistic filtering becomes tractable for discrete state spaces. The filtering is then performed by the forward algorithm given the instantaneous sequential RSSI measurements. The performance of the system is evaluated by taking the mean squared errors of the most probable positions at each time step to their corresponding ground-truth positions. We report the statistics of the error distributions and see that we achieve promising results. The approach is also finally evaluated by its runtime and memory usage.

Nowadays, various AI applications based on Convolutional Neural Networks (CNNs) are widely deployed on GPU-accelerated devices. However, due to the lack of visibility into GPU internal scheduling, accurately modeling the performance of CNN inference tasks or estimating the latency of CNN tasks that are executing or waiting on the GPU is challenging. This hurts the multi-model scheduling on multi-device and CNN real-time inference. Therefore, in this paper, we propose a time estimation method to estimate the forward execution time of a convolutional layer with an arbitrary shape on a GPU. The proposed method divides an explicit General Matrix Multiplication (GEMM) convolution operation into a series of estimatable GPU operations and constructs performance models at the level of sub-operations rather than hardware instructions or entire models. Also, the proposed method can be easily adapted to different hardware devices or underlying algorithm implementations, since it focuses on the variation of execution time relative to the input data scale, without focusing on specific instructions or hardware actions. According to the experiments on four typical CUDA compatible platforms, the proposed method has an average error rate of less than 5% for convolutional layers in some practical CNN models, and has about 8% error rate in estimating GEMM convolution implementations provided by cuDNN library. Experiments show that the proposed method can predict the forward execution time of convolutional layers of arbitrary size in CNN inference tasks on different GPU models.

We consider the problem of service hosting where a service provider can dynamically rent edge resources via short term contracts to ensure better quality of service to its customers. The service can also be partially hosted at the edge, in which case, customers’ requests can be partially served at the edge. The total cost incurred by the system is modeled as a combination of the rent cost, the service cost incurred due to latency in serving customers, and the fetch cost incurred as a result of the bandwidth used to fetch the code/databases of the service from the cloud servers to host the service at the edge. In this paper, we compare multiple hosting policies with regret as a metric, defined as the difference in the cost incurred by the policy and the optimal policy over some time horizon $T$. In particular we consider the Retro Renting (RR) and Follow The Perturbed Leader (FTPL) policies proposed in the literature and provide performance guarantees on the regret of these policies. We show that under i.i.d stochastic arrivals, RR policy has linear regret while FTPL policy has constant regret. Next, we propose a variant of FTPL, namely Wait then FTPL (W-FTPL), which also has constant regret while demonstrating much better dependence on the fetch cost. We also show that under adversarial arrivals, RR policy has linear regret while both FTPL and W-FTPL have regret $O\left(\sqrt{T}\right)$ which is order-optimal.