Predicting Which Category Something Belongs To
From describing patterns to predicting outcomes — will this customer churn? Will this loan default? Which product will this customer buy?
Sets 01–05 described, diagnosed, forecasted, and explained numerical outcomes. Set 08 tackles a different kind of question: predicting a category. Will this customer leave or stay? Is this transaction fraudulent or legitimate? Which of three products will this prospect choose? These are classification problems — and they demand different techniques.
Area Under ROC Curve
technique
Logistic Regression (Binary)
Models the probability of a binary outcome (0 or 1) using a logistic curve. Produces probabilities and odds ratios — highly interpretable.
Logistic Regression (Multinomial)
Extension to three or more outcome categories. Each category is compared against a reference group. Common in market segmentation and product choice modelling.
Decision Trees (CART)
Builds a tree of if-then rules by recursively splitting data on the most informative variable. Highly interpretable — the rules can be presented directly to business stakeholders.
Random Forest
An ensemble of many decision trees, each trained on a random sample of data and features. The majority vote determines the prediction. More accurate and less prone to overfitting than a single tree.
Naïve Bayes
Probabilistic classifier based on Bayes' theorem. Assumes features are independent (the "naïve" assumption). Fast, effective for text classification and spam detection.
Support Vector Machine (SVM)
Finds the optimal boundary (hyperplane) between classes that maximises the margin between classes. Powerful in high-dimensional spaces like text or image data.
Gradient Boosting (XGBoost)
Builds trees sequentially — each new tree corrects the errors of the previous ones. Currently the highest-performing algorithm on most structured business datasets.
Model Evaluation
Confusion matrix, accuracy, precision, recall, F1-score, and ROC-AUC. The framework for understanding how well — and in what ways — a classifier performs.
Regression (Sets 03–04) predicts a continuous number: "How much will revenue be?" Classification predicts a discrete category: "Will this customer churn — yes or no?" The same predictors can be used in both, but the algorithms, loss functions, and evaluation metrics are entirely different. Never use regression to predict a binary outcome — use logistic regression or a classification algorithm instead.
Logistic Regression
The most interpretable classification technique. Produces probabilities, odds ratios, and clear coefficient tables that non-technical stakeholders can understand.
Logistic regression predicts the probability that an observation belongs to a category. Instead of fitting a straight line (which could predict probabilities below 0 or above 1), it fits a logistic (S-shaped) curve that always stays between 0 and 1.
Output is always between 0 and 1 — a probability. Decision threshold (typically 0.5) converts probability to a predicted class.
Interpreting logistic regression output — odds ratios
Unlike linear regression where β is the change in Y per unit of X, in logistic regression the exponentiated coefficient (e^β) is the odds ratio — how much the odds of the outcome multiply per unit increase in X.
| Odds Ratio (e^β) | Meaning | Business interpretation |
|---|---|---|
| OR = 1.0 | No effect — odds unchanged | This variable makes no difference to the probability of the outcome |
| OR = 1.5 | Odds 50% higher | Each unit increase in X multiplies the odds of churning by 1.5 (50% more likely) |
| OR = 2.0 | Odds double | The outcome is twice as likely per unit increase in X — a strong positive predictor |
| OR = 0.7 | Odds 30% lower | Each unit increase in X reduces the probability of the outcome — a protective factor |
| OR = 0.3 | Odds 70% lower | Strong negative predictor — substantially reduces the likelihood of the outcome |
Binary logistic regression predicting churn (1=churned, 0=stayed) with three predictors:
| Predictor | β | Odds Ratio (e^β) | p-value | Interpretation |
|---|---|---|---|---|
| Contract Length (months) | −0.08 | 0.92 | 0.001 | Each additional month of contract reduces churn odds by 8% |
| Support Tickets (last 3mo) | 0.41 | 1.51 | 0.003 | Each additional ticket increases churn odds by 51% |
| Price Sensitivity Score | 0.29 | 1.34 | 0.012 | Higher price sensitivity increases churn odds by 34% |
Business action: Customers with short contracts, multiple recent support issues, and high price sensitivity are the highest-risk segment. Target them with proactive outreach before contract renewal.
When the outcome has 3+ categories (e.g. "Premium," "Standard," "Budget" tier chosen), use multinomial logistic regression. SPSS: . Each category is compared against a reference group (e.g. "Standard" is the reference, and the model estimates the odds of choosing "Premium" vs Standard, and "Budget" vs Standard). Python: .
Decision Trees
If-then rules that any stakeholder can understand. The most explainable classification technique.
A decision tree works by repeatedly splitting the data on the variable (and value) that best separates the classes. The result is a tree of if-then rules: "IF contract length < 12 months AND support tickets > 3 THEN predict churn." Each path from root to leaf is a complete classification rule.
The tree produces four segments with very different default probabilities: 82% (high risk), 41% and 28% (medium risk), 6% (approve confidently). A credit officer can apply these rules manually — no software required for a decision.
Key decision tree concepts
| Concept | Plain explanation | Why it matters |
|---|---|---|
| Gini Impurity | Measures how mixed the classes are in a node. Gini=0 means perfectly pure (all one class). Gini=0.5 means maximum mixing. | CART algorithm splits on the variable that most reduces Gini — creating purer child nodes |
| Information Gain | How much a split reduces entropy (uncertainty about class membership). Higher gain = better split. | Alternative to Gini — used in ID3/C4.5 algorithms |
| Max Depth | Maximum number of levels in the tree. Shallow trees (depth 3–5) are interpretable. Deep trees overfit. | The most important hyperparameter to control. Start with depth=3 and increase gradually |
| Pruning | Removing branches that don't add predictive power. Prevents overfitting by simplifying the tree. | A tree that perfectly fits training data will often fail on new data — pruning improves generalisation |
| Overfitting | When the tree memorises training data rather than learning patterns. Produces high training accuracy but poor test accuracy. | Always evaluate on a held-out test set. A large gap between train and test accuracy = overfitting |
Random Forest
The wisdom of many trees. More accurate than a single decision tree — and more robust to noise and outliers.
A random forest builds hundreds (or thousands) of decision trees, each trained on a random sample of rows (with replacement — "bagging") and a random subset of features at each split. Every tree makes a prediction; the final prediction is the majority vote (classification) or average (regression) across all trees.
Why random forests outperform single trees
| Property | Single Decision Tree | Random Forest |
|---|---|---|
| Accuracy | Moderate — limited by depth/pruning trade-off | Higher — averaging reduces variance |
| Overfitting | High risk — especially deep trees | Low risk — randomness in sampling prevents trees from memorising |
| Interpretability | High — rules can be stated explicitly | Low — hundreds of trees cannot be summarised as rules |
| Feature importance | Available but unstable (changes with tree structure) | Stable — averaged across all trees; highly reliable for driver identification |
| Speed | Very fast | Slower — but parallelisable; modern hardware handles 1,000 trees easily |
Random forests produce a feature importance ranking: which variables were most useful for reducing error across all trees. This is one of the most reliable driver-identification tools available. Unlike standardised regression coefficients, feature importance handles non-linear relationships and interactions automatically. Use it to identify the top 5–10 predictors before building a simpler, more interpretable model.
n_estimators — number of trees. More trees = more stable, diminishing returns after 200–500. max_depth — maximum depth of each tree. Shallow trees (3–7) reduce overfitting. max_features — number of features considered at each split. Typically √(total features) for classification. min_samples_leaf — minimum observations required in a leaf. Higher values produce simpler trees.
Gradient Boosting & XGBoost
Sequential learning — each model corrects the mistakes of the previous one. Currently the highest-performing algorithm on most structured business datasets.
How boosting works
Unlike random forests (which build trees in parallel and average them), boosting builds trees sequentially. Each new tree focuses on the observations the previous trees got wrong — fitting the residuals (errors) rather than the original targets. The final prediction is the weighted sum of all trees.
XGBoost — why it dominates
XGBoost (eXtreme Gradient Boosting) is an optimised implementation of gradient boosting with additional regularisation terms that prevent overfitting. It has won more Kaggle competitions and industrial prediction challenges than any other single algorithm.
| Algorithm | Type | When to use | Interpretability |
|---|---|---|---|
| Logistic Regression | Single model | When interpretability is essential; regulatory environments; small datasets | Very high — odds ratios per variable |
| Decision Tree | Single model | When stakeholders need explicit rules; first-pass exploration | High — rules stated directly |
| Random Forest | Ensemble (parallel) | Good default for most classification problems; feature importance needed | Low — "black box" but feature importance available |
| XGBoost | Ensemble (sequential) | Maximum predictive accuracy; structured/tabular data; competition benchmarks | Low — requires SHAP values for explanation |
| Naïve Bayes | Probabilistic | Text classification; spam detection; when features are genuinely independent | Moderate — probabilities per feature value |
| SVM | Geometric | High-dimensional data; text; image classification | Low — hyperplane in high-dimensional space |
SHAP (SHapley Additive exPlanations) decomposes each individual prediction into contributions from each feature. For any single customer, you can say: "This customer was predicted to churn because: support tickets (+0.32), short contract (+0.21), high price sensitivity (+0.18), offset by long tenure (−0.15)." In Python: → . SHAP values make even XGBoost auditable for regulatory compliance.
Model Evaluation
Accuracy is rarely enough. The right metric depends on the business cost of different types of errors.
Classification models make two types of errors: predicting positive when it is actually negative (False Positive) and predicting negative when it is actually positive (False Negative). In most business problems these errors have very different costs — and the right metric reflects which error matters more.
The Confusion Matrix
The foundation of all classification metrics. A confusion matrix cross-tabulates actual vs predicted classes.
Key metrics derived from the confusion matrix
| Metric | Formula | Business interpretation | Use when |
|---|---|---|---|
| Accuracy | TP+TN / Total | % of all predictions that are correct | Classes are balanced — misleading with rare events (1% fraud: predict all legitimate and get 99% accuracy) |
| Precision | TP / (TP+FP) | Of all customers flagged as churners, what % actually churn? | False positives are costly — e.g. expensive retention outreach sent to customers who weren't going to leave |
| Recall (Sensitivity) | TP / (TP+FN) | Of all actual churners, what % did we catch? | False negatives are costly — e.g. missing a fraudulent transaction that causes customer loss |
| F1 Score | 2 × (Prec×Recall) / (Prec+Recall) | Harmonic mean of precision and recall — balances both | Both types of error matter; want a single balanced metric |
| ROC-AUC | Area under ROC curve | Probability that the model ranks a random positive higher than a random negative. AUC=0.5 = random; AUC=1.0 = perfect | Overall model discrimination quality — best for comparing models |
Lowering the decision threshold (e.g. from 0.5 to 0.3) means more customers are flagged as churners — recall increases (you catch more real churners) but precision decreases (more false alarms). Raising the threshold means fewer alarms — precision rises but recall falls. The optimal threshold depends on the relative cost of false positives and false negatives in your specific business context.
| Use case | Costly error | Key metric | Why |
|---|---|---|---|
| Cancer screening | Missing cancer (FN) | Recall / Sensitivity | Missing a positive is far worse than a false alarm |
| Spam filter | Blocking legitimate email (FP) | Precision | Blocking important emails annoys users; spam is tolerable |
| Fraud detection | Missing fraud (FN) | Recall + AUC | Fraudulent transactions cause real financial loss |
| Retention campaign | Wasting budget on stayers (FP) | Precision | Outreach is expensive — target only likely churners |
| Loan approval | Approving defaulters (FP) | Precision | Bad loans are costly; rejected good applicants go elsewhere |
SPSS & RapidMiner
GUI-based classification — logistic regression in SPSS, full ML pipelines in RapidMiner. No code required.
RapidMiner uses a drag-and-drop workflow canvas. Each operator (Import, Model, Evaluate) is connected visually. No code required — ideal for non-programmers who need full ML capability.
Python — scikit-learn & XGBoost
The complete classification pipeline — from preprocessing through model training, evaluation, and explanation.
Three Simulators
Build intuition for confusion matrices, classification thresholds, and feature importance through live interaction.
Simulator 1 · Confusion Matrix Builder
A churn model has been applied to 1,000 customers. Adjust the counts of TP, FP, FN, TN and watch how accuracy, precision, recall, and F1 change in real time.
Simulator 2 · Decision Threshold Explorer
The model produces a probability for each customer. The threshold determines who is classified as "churn." Move the threshold and watch precision and recall trade off against each other.
Simulator 3 · Feature Importance Interpreter
A random forest has been trained on telecom churn data. The feature importance scores are shown. Rank the business actions from most to least impactful based on which features drive churn.
Common Mistakes
Classification mistakes can cost organisations real money — through missed risks or wasted interventions.
Using accuracy as the primary metric on imbalanced data
A model that predicts "no fraud" for every transaction achieves 99.5% accuracy on a dataset with 0.5% fraud — but catches zero fraudulent transactions. Always check class balance first. On imbalanced data, use AUC, precision, recall, and F1 instead.
Evaluating on training data
A model evaluated on the same data it was trained on will always look better than it really is — it has memorised the answers. Always use a held-out test set or cross-validation. A large gap between train accuracy and test accuracy = overfitting.
Ignoring the business cost of different error types
Defaulting to a 0.5 threshold without considering whether false positives or false negatives are more costly is an analytical failure. Always ask: what does it cost to flag a loyal customer as a churner? What does it cost to miss a churner?
Data leakage
Including variables in the model that would not be available at prediction time — for example, using "number of days since last call" in a churn model when you would not know this for future customers. Leakage produces unrealistically high accuracy that collapses in production.
Choosing complexity over interpretability without justification
XGBoost has marginally higher AUC than logistic regression — but logistic regression is explainable, auditable, and deployable in regulated environments. The most accurate model is not always the right model. Match complexity to the organisation's needs.
Not re-training as the world changes
A churn model trained in 2022 may be unreliable in 2025 if customer behaviour has shifted. Classification models degrade as the real world drifts away from the training data. Monitor model performance on new data monthly and retrain when accuracy drops.