Finding the Best Box-Cox Transformation in Big Data with Meta-Model Learning: A Case Study on QCT Developer Cloud

Abstract

Finding the best model to reveal potential relationships of a given set of data is not an easy job and often requires many iterations of trial and errors for model sections, feature selections and parameters tuning. This problem is greatly complicated in the big data era where the I/O bottlenecks significantly slowed down the time needed to finding the best model. In this article, we examine the case of Box-Cox transformation when assumptions of a regression model are violated. Specifically, we construct and compute a set of summary statistics and transformed the maximum likelihood computation into a per-role operational fashion. The innovative algorithms reduced the big data machine learning problem into a stream based small data learning problem. Once the Box-Cox information array is obtained, the optimal power transformation as well as the corresponding estimates of model parameters can be quickly computed. To evaluate the performance, we implemented the proposed Box-Cox algorithms on QCT developer cloud. Our results showed that by leveraging both the algorithms and the QCT cloud technology, find the fittest model from 101 potential parameters is much faster than the conventional approach.