Churn Modeling for Mobile Telecommunications (PDF, 298K)
Introduction
The Duke University/NCR Teradata Center for Customer Relationship management (CRM) is devoted to advancement of CRM and supports research and education in both academic and practitioner settings. One of the center's core objectives is to "merge the best of theory and practice" in CRM and it offers a variety of educational and research forums for both students and management in global corporations.
In 2003 the Center sought to identify the best predictive modeling techniques to help manage a vexing wireless telecommunications problem: customer churn. While other industries are also faced with customers who defect to competitors, at the retail level, wireless customers switch service providers at a rate of about 25% per year, or 25 per month. In the early 1990s when new subscriber growth rates were in the 50% range, telecommunications companies were tempted to focus on new acquisition rather than on customer retention. However, in a new era of much slower growth rates as low as 10%, it is becoming evident that customer retention is vital to overall profitability. The key to customer retention is predicting which customers are most at risk of defecting to a competitor and offering the most valuable of them incentives to stay. To execute such a strategy effectively one must be able to develop highly accurate predictions or churn scorecards so that the retention effort is focused on the relevant customers.
The Data
The data were provided by a major wireless telecommunications company using its own customer records for the second half of 2001. Account summary data was provided for 100,000 customers who had been with the company for at least 6 months. To assist in the modeling process the churners were oversampled so that one half of the sample consisted of churners (those who left the company by the end of the following 60 days) and the other half were customers remaining with the company at least another 60 days. A broad range of 171 potential predictors were made available, spanning all the types of data a typical service provider would routinely have available. Predictor data included:
- Demographics: Age, Location, Number and ages of children, etc.
- Financial: Credit score, Credit card ownership
- Product details: Handset price, Handset capabilities, etc.
- Phone usage: Number and duration of various categories of calls, etc.
Evaluation Criteria
The "training" or "calibration" data described above were provided to support predictive modeling development. Participants were asked to use their best models to predict the probability of churn for two different groups of customers to be scored: a "current" sample of 51,306 drawn from the latter half of 2001 and a "future" sample of 100,462 customers drawn from the first quarter of 2002. Predicting "future" data is generally considered more difficult because external factors and behavioral patterns may change over time. Of course in real world settings predictive models are always applied to future data and the tournament organizers wanted to reproduce a similar context.
Each contestant in the tournament was asked to rank the current and future score samples in descending order by probability of churn. Using the actual churn status available to the tournament organizers two performance measures were calculated for each predictive model: the overall Gini measure and the lift in the top decile. The two measures were calculated for the two samples, current and future, so that there were four performance scores available for every contestant. The evaluation criteria are described in detail in a number of locations including the tournament web site. The top-decile lift is the easiest to explain non-technically: it measures the number of actual churners captured among the customers ranked most likely to churn by a model.
Results
Contestants were free to develop a separate model for each measure if they wished to try to optimize their models both to either the time period or the evaluation criterion, or both. Salford Systems submitted two models: a straightforward out-of-the-box TreeNet® ® model, and a more complicated model averaging the predictions of several different TreeNet® models. The contest results are summarized below along with an explanation of their meaning and significance.
| Data Set | Measure | TreeNet® Ensemble | Single TreeNet® | 2nd Best | Avg. (Std) |
| Current | Top Decile Lift | 2.90 | 2.88 | 2.80 | 2.14 (.536) |
| Current | Gini | .409 | .403 | .370 | .269 (.096) |
| Future | Top Decile Lift | 3.01 | 2.99 | 2.74 | 2.09 (.585) |
| Future | Gini | .400 | .403 | .361 | .261 (.098) |
In the "Current" data set the contestants were provided with account data for 100,462 customers of which 1,808 churned in the following month, at a rate of approximately 1.8% per month. Of course the tournament contestants did not know which accounts churned. If 10% of all accounts were chosen at random we would expect to capture 10% or 181 churners. The TreeNet® ensemble method captured 525 and a single routine TreeNet® model captured 521. by contrast, the best competing model captured 507 and the average model captured 387. Assuming a mobile telecommunications service provider had 1 million accounts, using the Salford model would capture an additional 1,380 accounts per month over a routine model.
In the "Future" data set there were 51,036 accounts of which 924 churned, also at a 1.8% per month rate. A 10% portion of this data selected at random should capture about 92 churners. For this data the ensemble TreeNet® model captured 278 and single TreeNet® captured 276. The best alternative model captured 253 churners and the average model captured 193 churners. In a one million account portfolio the Salford Systems model would capture about 500 more churners per month than the best alternative model and 850 more churners per month than the average model.
The benefits of the TreeNet® model in the top decile have two components. First, by selecting the right accounts to target, marketing resources are not wasted on the wrong customers. Second, the TreeNet® model would allow the company to achieve a higher retention rate. There is more to the model than the top decile however. Given that the TreeNet® model is superior across the board, as reflected in the Gini coefficients, the use of the TreeNet® model in a targeted retention plan should yield an additional 20,000 plus churners identified pear year.
Further advantages stem from the ease with which the TreeNet® model is constructed. Since the process is fully automatic TreeNet® models can be built in less time and with less preparatory effort than competing methods.
See also TreeNet® Wins Major Competition - Sweeps All Four Categories