Cumulative Probability Distribution Model for Evaluating User Behavior Prediction Algorithms

Abstract

User behavior analysis and prediction has been widely applied in personalized search, advertising precise delivery and other personalized services. It is a core problem how to evaluate the performance of prediction models or algorithms. The most used off-line experiment is a simple and convenient evaluation strategy. However, the existing assessment measures are most based on arithmetic average value theory, such as precision, recall, F measure, mean absolute error (MAE), root mean squared error (RMSE) etc. These approaches have two drawbacks. First, they cannot depict the prediction performance within a more fine-grained view and they only provide one average value to compare different algorithms' performances. Second, they are not reasonable if the evaluation results are not follow normal distribution. In this paper, according to analyze a mass of prediction evaluation results, we find that some performance evaluation results follow approximate power low distribution but not normal distribution. Therefore, the paper proposes a cumulative probability distribution model to evaluate the performance of prediction algorithms. The model first calculates the probability of each evaluation results. And then, it depicts the cumulative probability distribution function. Moreover, we further present an evaluation expectation value (EEV) to represent the overall performance of the prediction algorithms. Experiments on two real data sets show that the proposed model can provide deeper and more accurate assessment results.