Web Services Wind Tunnel: On Performance Testing Large-Scale Stateful Web Services

New versions of existing large-scale web services such as Passport.comcopy have to go through rigorous performance evaluations in order to ensure a high degree of availability. Performance testing (such as benchmarking, scalability, and capacity tests) of large-scale stateful systems in managed test environments has many different challenges, mainly related to the reproducibility of production conditions in live data centers. One of these challenges is creating a dataset in a test environment that mimics the actual dataset in production. Other challenges involve the characterization of load patterns in production based on log analysis and proper load simulation via reutilization of data from the existing dataset. The intent of this paper is to describe practical approaches to address some of the aforementioned challenges through the use of various novel techniques. For example, this paper discusses data sanitization, which is the alteration of large datasets in a controlled manner to obfuscate sensitive information, preserving data integrity, relationships, and data equivalence classes. This paper also provides techniques for load pattern characterization via the application of Markov chains to custom and generic logs, as well as general guidelines for the development of cache-based load simulation tools tailored for the performance evaluation of stateful systems.