Building and validating a reduced TPC-H benchmark

Abstract

The properties of computer system such as performance, energy, reliability, etc. are commonly evaluated by running benchmarks. However, the benchmarking process is complicated to set-up and use and running the benchmarks takes a substantial amount of time. Furthermore, when designing a computer, architects resort to simulation of the system, increasing the benchmarking time by several orders of magnitude. This problem can be alleviated by reducing the execution time of the benchmark suite. In this paper, we investigate a method to reduce the number of queries in TPC-H, a decision support system benchmark. Our evaluation shows that out of the 22 original queries, a subset of only 6 achieves a high representativeness at only 40% of the original execution time. The results also show that subsets built for a particular database size may be used for evaluating computer systems with other database sizes. Finally, we validate our approach against the full benchmark execution for a set of architecture case studies. The validation results show that the proposed subsets lead to similar conclusions than the ones drawn using the full benchmark. © 2006 IEEE.