How to Interpret Model Performance with Cost Functions
In this 10-part video series we discuss the concept of cost functions, which are directly related to the performance of data mining and predictive models.
Part 1: An Introduction to Understanding Cost Functions
This video is the first of 10 in a series that will help you, the analyst, understand the importance of cost functions, how they are used, and how the relate to model performance. In this 12-minute video we will introduce:
- The supervised learning problem: what is it and how is it applied in machine learning?
- How cost functions are used to solve the supervised learning problem
- Evaluating the ‘fit’ of the current response surface on the data available
- What is special about predicting the response in a regression problem?
Part 2: Least Squares Deviation Cost for a Regression Problem
In this 11-minute video we will cover:
- What is the Least Squares Deviation Cost function?
- What are the advantages of LS?
- An introduction to the underpinnings of the LSD’s statistical properties on the formula level
- What are the disadvantages of LS?
Part 3: Least Absolute Deviation and Huber-M Cost for a Regression Problem
In this 14-minute video we conclude our analysis of cost functions that can be applied to a regression problem. The outline for this video is as follows:
- What is the Least Absolute Deviation Cost function?
- How is LAD different from LS (part2)?
- Understanding of how LAD handles outliers
- What are LAD’s negative attributes?
- What is Huber-M Cost, and how does it compare to LS and LAD?
- Conclusion to cost functions used for a regression problem
Part 4: Introducing the Binary Classification Problem
In this 10-minute video we will quickly introduce the binary classification problem. This will set the stage for a later discussion of the various cost functions that can be applied to this type of problem.
- Why binary classification is commonly used among data analysts
- What are the fundamentals of the binary classification problem
- How to construct a simple response surface using linear regression
- How to use decision rules to make predictions
Part 5: Evaluating Prediction Success with Precision and Recall
In this 13-minute video we will begin our analysis of cost functions for binary classification. We will hone in on the following topics:
- Recap: working with a binary classification problem
- Evaluating the ‘positive’ group of predictions
- Define: precision, recall (sensitivity), and specificity
Part 6: Measuring Performance with the ROC Curve
In this 19-minute video we will review:
- The prediction success table
- Sensitivity vs. Specificity
- What is the ROC Curve, and how is it used to evaluate model performance?
- Advantages of evaluating based on ROC
- How to utilize the Area Under Curve (AUC)
Part 7: Assessing Model Performance with Gains and Lift
In this 20-minute video we will explore how to evaluate the performance of a binary classification model using gains and lift charts. In this data mining tutorial we use a direct marketing example to explain the concept of gains and lift; which is essentially a plot that represents sensitivity vs. support in the example marketing campaign.
Part 8: Direct Interpretation of Response with Logistic Function
In this 19-minute video we will cover:
- The mathematical structure behind the algorithm
- Direct interpretation of likelihood as a cost function
- Two ways to write logistic cost function
Part 9: Multinomial Classification- Expected Cost
In this 23-minute video we will discuss:
- What is multinomial classification?
- Assigning prior probabilities to your model
- Interpreting the expected cost
- Define: base-line, relative cost, and unit costs, and
- Alternative approaches to finding the expected cost
Part 10: Multinomial Classification- Log Likelihood
This video concludes our series on cost functions. We will:
- Build a probability response model that predicts probabilities
- Evaluate the performance of your multinomial classification model with Log-Likelihood
- Applying Log Likelihood to ensemble modeling scenarios, and discover an
- Alternative method: Margin-based cost function