Benchmarking operating system dependability: Windows 2000 as a case study

Abstract

We propose a dependability benchmark suitable for a general purpose operating system (OS). The specifications of the benchmark components are presented and illustrated on a benchmark prototype dedicated to Windows 2000. The important novelty, as regards OS dependability benchmarking, is threefold. First, it lies on a comprehensive and structured set of measures: outcomes are considered both at the OS level and at the application level. Second, these measures include not only robustness measures (e.g., the distribution among the observed outcomes for the OS and the application: error codes, exceptions, workload correct or erroneous completion, OS and application hang), but also the related temporal measures in the presence of faults (e.g., system call and workload execution times, as well as operating system restart time). Finally, we are considering a realistic workload (namely, TPC-C client), instead of a synthetic workload.