How to Interpret Model Performance with Cost Functions
In this 10-part video series we discuss the concept of cost functions, which are directly related to the performance of data mining and predictive models.
Part 1: An Introduction to Understanding Cost Functions
- The supervised learning problem: what is it and how is it applied in machine learning?
- How cost functions are used to solve the supervised learning problem
- Evaluating the ‘fit’ of the current response surface on the data available
- What is special about predicting the response in a regression problem?
Part 2: Least Squares Deviation Cost for a Regression Problem
- What is the Least Squares Deviation Cost function?
- What are the advantages of LS?
- An introduction to the underpinnings of the LSD’s statistical properties on the formula level
- What are the disadvantages of LS?
Part 3: Least Absolute Deviation and Huber-M Cost for a Regression Problem
- What is the Least Absolute Deviation Cost function?
- How is LAD different from LS (part2)?
- Understanding of how LAD handles outliers
- What are LAD’s negative attributes?
- What is Huber-M Cost, and how does it compare to LS and LAD?
- Conclusion to cost functions used for a regression problem
Part 4: Introducing the Binary Classification Problem
- Why binary classification is commonly used among data analysts
- What are the fundamentals of the binary classification problem
- How to construct a simple response surface using linear regression
- How to use decision rules to make predictions
Part 5: Evaluating Prediction Success with Precision and Recall
- Recap: working with a binary classification problem
- Evaluating the ‘positive’ group of predictions
- Define: precision, recall (sensitivity), and specificity
Part 6: Measuring Performance with the ROC Curve
In this 19-minute video we will review:
- The prediction success table
- Sensitivity vs. Specificity
- What is the ROC Curve, and how is it used to evaluate model performance?
- Advantages of evaluating based on ROC
- How to utilize the Area Under Curve (AUC)
Part 7: Assessing Model Performance with Gains and Lift
Part 8: Direct Interpretation of Response with Logistic Function
- The mathematical structure behind the algorithm
- Direct interpretation of likelihood as a cost function
- Two ways to write logistic cost function
Part 9: Multinomial Classification- Expected Cost
- What is multinomial classification?
- Assigning prior probabilities to your model
- Interpreting the expected cost
- Define: base-line, relative cost, and unit costs, and
- Alternative approaches to finding the expected cost
Part 10: Multinomial Classification- Log Likelihood
- Build a probability response model that predicts probabilities
- Evaluate the performance of your multinomial classification model with Log-Likelihood
- Applying Log Likelihood to ensemble modeling scenarios, and discover an
- Alternative method: Margin-based cost function