User Tools

Site Tools


bicn01:dm06

Testing Data Mining Models

Now that Susan has processed the model by using the targeted mailing scenario training set, he will test your models against the testing set. Because the data in the testing set already contains known values for bike buying, it is easy to determine whether the model's predictions are correct. The model that performs the best will be used by the AdventureBikes marketing department to identify the customers for their targeted mailing campaign.

:!: Validation is an important step in the data mining process. Knowing how well your targeted mailing mining models perform against real data is important before you deploy the models into a production environment.

On the Mining Accuracy Chart tab of Data Mining Designer, you can calculate how well each of your models makes predictions, and compare the results of each model directly against the results of the other models. This method of comparison is referred to as a Lift Chart. Typically, the predictive accuracy of a mining model is measured by either lift or classification accuracy.

A lift chart plots the results of prediction queries from a testing dataset against known values for the predictable column that exist in the dataset. The chart displays the results of the mining model, together with a representation of the results that an ideal model would produce, and a representation of the results of random guessing. Any improvement over the random line is called lift. The more lift that the model demonstrates, the more effective the model is. Only mining models that contain discrete predictable attributes can be compared in a lift chart.

Other validation methods are

  • Profit Charts: After entering facts related to costs, such as mailing fees, Analysis Services displays a curve that shows the lift provided by the model, and also calculates the return on the investment when the model is used.
  • A scatter plot is generated instead of a lift chart whenever the predictable attributes has continuous values.
  • A classification matrix shows the actual count of true positives, false positives, true negatives, and false negatives for each predictable attribute.
  • A Cross-Validation Report divides a data set into multiple cross-sections, automatically creates and trains multiple models on the subsets, and then calculates accuracy for all the models. By reviewing the statistics that are generated, you can assess how well a model generalizes across different data sets, or determine which of several models on a structure performs the best.

Testing Accuracy with Lift Charts

On the Mining Accuracy Chart tab of Data Mining Designer, you can calculate how well each of your models makes predictions, and compare the results of each model directly against the results of the other models. This method of comparison is referred to as a lift chart.

The first step in testing the accuracy of your mining models is to select the data source that you will use for testing. You will test how well the models perform against your testing data and then you will use them with external data.

Separating data into training and testing sets is an important part of evaluating data mining models. Typically, when you partition a data set into a training set and testing set, most of the data is used for training, and a smaller portion of the data is used for testing.
  • Switch to the Mining Accuracy Chart tab in Data Mining Designer and select the Input Selection tab.

  • In the Select data set to be used for Accuracy Chart group box, select Use mining model test cases to test your models by using the testing data that you set aside when you created the mining model.

The next step is to select the models that you want to include in the lift chart, the predictable column against which to compare the models, and the value to predict.

  • In the Predictable Column Name column, verify that Bike Buyer Flag is selected for each model.

By default, all the models in the mining structure are selected. You can decide not to include a model, but for this tutorial leave all the models selected..

  • In the Predict Value column, select “yes” (BikeBuyers). The same value is automatically filled in for each model that has the same predictable column.
  • Select the Lift Chart tab to display the lift chart.
The marketing department at AdventureBikes wants to create a targeted mailing campaign. From past campaigns, they know that a 10 percent response rate is typical. They have a list of 10,000 potential customers stored in a table in the database. Therefore, based on the typical response rate, they can expect 1,000 of the potential customers to respond.

However, the money budgeted for the project is not enough to reach all 10,000 customers in the database. Based on the budget, they can afford to mail an advertisement to only 5,000 customers. The marketing department has two choices:

  • Randomly select 5,000 customers to target
  • Use a mining model to target the 5,000 customers who are most likely to respond.

If the company randomly selects 5,000 customers, they can expect to receive only 500 responses, based on the typical response rate. This scenario is what the random line in the lift chart represents. However, if the marketing department uses a mining model to target their mailing, they can expect a larger response rate because they can target those customers who are most likely to respond. If the model is perfect, it means that the model creates predictions that are never wrong, and the company could expect to receive 1,000 responses by mailing to the 1,000 potential customers recommended by the model. This scenario is what the ideal line in the lift chart represents. The reality is that the mining model most likely falls between these two extremes; between a random guess and a perfect prediction. Any improvement from the random guess is considered to be lift.

When you click the tab, a prediction query runs against the server and database for the mining structure and the input table or test data. The results are plotted on the graph.

When you enter a Predict Value in the Input Selection, the lift chart plots a Random Guess Model as well as an Ideal Model.

  • The x-axis of the chart represents the percentage of the test dataset that is used to compare the predictions.
  • The y-axis of the chart represents the percentage of predicted values.

In this chart, the target attribute is Bike Buyer and the target value is “yes”, meaning that the customer purchased a bike or is likely to do so.

You can tell from the chart that the ideal line peaks at around 42 percent (1), meaning that if you had a perfect model, you could reach 100 percent of your targeted customers by sending a mailing to only 42 % of the total customers. The actual lift for the STM-Decision-Tree when you target around 30 percent of the population is about 73 percent (2), meaning you could reach about 70 percent of your targeted customers by sending the mailing to 30 percent of the total customer population.

The Mining Legend contains the actual values at any point on the curves.

From these results, you can see that, when measured at 29,70% of all cases (Population percentage), the STM-Decision-Tree model can predict the bike buying behavior of 63,84% of the target population. In other words, if you sent out a targeted mailing to only 30 percent of the prospective customers in your database, you could reach slightly more than two-third of your target audience. If you used the STM-Clustering model, you could reach 49,10 percent of your targeted customers.

The value for Predict probability represents the threshold required to include a customer among the “likely to buy” cases. For each case, the model estimates the accuracy of each prediction and stores that value, which you can use to filter out or to target customers. For example, to identify the customers from the Decision-Tree model who are likely buyers, you would use a query to retrieve cases with a Predict probability of at least 76,95 percent.

The value for Score helps you compare models by calculating the effectiveness of the model across a normalized population. A higher score is better, so in this case you might decide that Decision Tree is really the most effective strategy.

bicn01/dm06.txt · Last modified: 2018/12/04 08:39 (external edit)