Finding Natural Groups in Your Data
No predefined categories. No outcome variable. Just the question: what natural structures exist in this data?
Sets 05 and 06 required a pre-defined outcome — a group to test, a class to predict. Set 09 is different. Segmentation and pattern discovery are unsupervised — there is no "correct answer" in the data to learn from. The algorithms find structure that the analyst did not specify in advance.
This is powerful but requires more judgement. The algorithm always finds clusters — whether they are real or coincidental. The analyst must decide whether the discovered structure is meaningful, stable, and actionable.
number of clusters
association rules
K-Means Clustering
Partitions data into K groups by minimising within-cluster variance. Fast and scalable. Requires specifying K in advance. Most widely used clustering algorithm.
Hierarchical Clustering
Builds a tree of nested clusters (dendrogram) without requiring K in advance. Agglomerative (bottom-up) is most common. Excellent for visualising structure and exploring natural groupings.
DBSCAN
Density-Based Spatial Clustering. Finds clusters of arbitrary shape and automatically identifies outliers as noise. No K required — number of clusters emerges from the data.
Cluster Profiling
Describing the characteristics of each cluster using summary statistics and visualisations. Turns mathematical output into business-actionable segment descriptions.
Association Rules (Apriori)
Finds co-occurrence rules in transactional data. "Customers who buy X also frequently buy Y." The foundation of market basket analysis and recommendation systems.
Market Basket Analysis
Specific application of association rules to purchase data. Identifies product combinations for cross-selling, promotions, and shelf placement decisions.
Elbow Method
Determines the optimal number of clusters K by plotting within-cluster sum of squares against K. The "elbow" point suggests where adding more clusters stops meaningfully improving fit.
Silhouette Analysis
Validates cluster quality. Silhouette score (−1 to +1) measures how similar each point is to its own cluster vs the nearest other cluster. Higher is better.
K-means will always produce exactly K clusters regardless of whether natural groups exist in the data. The algorithm has no way to tell you "there are no meaningful groups here." Always validate: do the clusters replicate on different samples? Do they predict outcomes the business cares about? Do they make business sense? Clustering is hypothesis-generating — not conclusion-confirming.
K-Means Clustering
The most widely used segmentation technique. Fast, scalable, and intuitive — but requires you to specify the number of clusters.
How K-Means Works
K-means uses Euclidean distance. If one variable is measured in thousands of dollars and another is a 1–5 rating, the dollar variable will dominate the clustering simply because of its larger scale. Always standardise all variables to mean=0 and standard deviation=1 before running K-means. In Python: . In SPSS: Z-scores transformation before clustering.
The Elbow Method — Choosing K
The elbow method runs K-means for K = 1, 2, 3 ... n and plots the Within-Cluster Sum of Squares (WCSS) — how compact the clusters are. As K increases, WCSS always decreases. The optimal K is where the curve bends — adding more clusters beyond this point gives diminishing returns.
A retailer uses K-means on 50,000 customers using three variables: total annual spend, purchase frequency, and average basket size. All three are standardised. The elbow plot shows a clear bend at K=4 — four natural customer segments emerge.
The four clusters are profiled and labelled: Cluster 1 (high spend, high frequency, large basket) = "Champions" · Cluster 2 (high spend, low frequency, very large basket) = "Big Occasion Shoppers" · Cluster 3 (medium spend, high frequency, small basket) = "Regular Convenience Shoppers" · Cluster 4 (low spend, low frequency, small basket) = "Occasional Browsers"
Each segment receives a different marketing strategy: Champions get loyalty rewards, Big Occasion Shoppers get event promotions, Regular Convenience Shoppers get subscription offers, Occasional Browsers get re-engagement campaigns.
| Property | K-Means behaviour | Implication |
|---|---|---|
| Shape | Finds spherical (round) clusters | Will fail on elongated, crescent-shaped, or interlocking clusters. Use DBSCAN for irregular shapes. |
| Outliers | Sensitive — outliers pull centroids toward them | Remove or cap extreme outliers before clustering. One extreme customer can distort an entire segment. |
| Initialisation | Random start = different results each run | Use K-means++ initialisation (default in Python/SPSS) and run multiple times. Take the run with lowest WCSS. |
| Scale | Scales to millions of records | The most practical algorithm for large business datasets. Results in minutes on modern hardware. |
Hierarchical Clustering & DBSCAN
Two alternatives when K-means falls short — flexible structure, no K required.
Hierarchical Clustering — the Dendrogram
Hierarchical clustering builds a tree diagram (dendrogram) that shows how observations and clusters merge as the similarity threshold increases. Unlike K-means, you do not need to specify K in advance — you can cut the tree at any level to produce any number of clusters.
| Linkage method | How distance between clusters is measured | When to use |
|---|---|---|
| Ward's linkage | Minimises the increase in total within-cluster variance when merging two clusters | Most commonly used. Produces compact, roughly equal-sized clusters. Default choice for business use. |
| Complete linkage | Distance between the two farthest points in each cluster | Produces tight, compact clusters. Sensitive to outliers. |
| Average linkage | Average distance between all pairs of points across the two clusters | Compromise between single and complete. Robust to outliers. |
| Single linkage | Distance between the two closest points across clusters | Good for elongated clusters. Prone to chaining — linking clusters through a single bridge point. |
A dendrogram for 200 employee satisfaction survey responses is produced using Ward's linkage. The vertical axis shows the distance (dissimilarity) at which clusters merge. By cutting the tree at a height of 15, the analyst obtains 3 clusters. By cutting at height 8, they get 5 clusters. The height at which you cut is a business decision — how many meaningful segments does your use case require?
A large "jump" in merge height between two levels suggests that the two clusters being merged are genuinely different. This jump identifies the natural number of clusters in the data.
DBSCAN — Density-Based Clustering
DBSCAN (Density-Based Spatial Clustering of Applications with Noise) finds clusters as dense regions of points separated by sparse regions. It requires two parameters: ε (epsilon) — the radius of the neighbourhood, and minPts — the minimum number of points required to form a dense region.
No K required
The number of clusters emerges from the data structure. If 3 dense regions exist, DBSCAN finds 3. If 7 exist, it finds 7. You do not need to guess K in advance.
Finds irregular shapes
Works on crescent, ring, or any arbitrary shape — K-means and hierarchical clustering assume compact, roughly spherical groups. DBSCAN does not.
Identifies noise
Points in sparse regions are labelled as noise (outliers) rather than forced into a cluster. In customer data this can identify genuinely unusual accounts worth separate investigation.
Sensitive to ε and minPts
Results change substantially with different parameter choices. Try multiple values and use silhouette scores to evaluate. Works poorly when clusters have very different densities.
| Algorithm | K required? | Shape | Outlier handling | Best for |
|---|---|---|---|---|
| K-Means | Yes — specify upfront | Spherical | Poor — outliers distort centroids | Large datasets, clear spherical groups, known K |
| Hierarchical | No — choose after | Flexible | Moderate | Small-medium datasets, exploratory segmentation, visual dendrograms |
| DBSCAN | No — emerges | Any shape | Excellent — labels noise explicitly | Irregular cluster shapes, anomaly detection, geospatial data |
Cluster Profiling
The algorithm finds clusters. You decide what they mean. Profiling turns numbers into actionable segment descriptions.
Cluster profiling is not a separate algorithm — it is the analytical discipline of characterising each cluster using descriptive statistics, visualisations, and domain knowledge. A cluster without a profile is just a number. A cluster with a profile is a business segment.
The Profiling Process
| Cluster | Avg Spend/mo | Avg Tenure | Support Tickets | Data Usage | Segment Name |
|---|---|---|---|---|---|
| 1 (n=4,200) | $95 | 48 mo | 0.3/mo | High | 🟢 Loyal High-Value |
| 2 (n=7,800) | $48 | 12 mo | 0.8/mo | Medium | 🟡 New At-Risk |
| 3 (n=5,100) | $72 | 36 mo | 0.2/mo | Low | 🔵 Stable Mid-Value |
| 4 (n=2,900) | $31 | 6 mo | 1.4/mo | Low | 🔴 High-Risk Churners |
Cluster 4 is the immediate priority: recent acquisition, low spend, high support volume — clear churn risk. Cluster 2 also warrants early intervention. Cluster 1 should receive loyalty rewards to maintain their high-value relationship.
A statistically clean cluster solution is not enough. Ask: Do the clusters predict something that matters? (e.g. Do they have different churn rates? Different lifetime values?) If the clusters do not separate on any meaningful outcome variable, the segmentation — however mathematically sound — may not be actionable for the business.
Association Rules & Market Basket Analysis
Finding what goes together. Which products are bought simultaneously — and how reliably?
Association rule mining discovers co-occurrence patterns in transactional data. The classic application is market basket analysis: "customers who buy bread also frequently buy butter." But the same technique applies to any dataset where items co-occur: website pages visited together, insurance products held simultaneously, diagnostic codes appearing in the same patient record.
Three Key Metrics
| Metric | Definition | Formula | Business interpretation |
|---|---|---|---|
| Support | How often the itemset (A and B together) appears in all transactions | P(A ∩ B) = transactions containing both / total transactions | Support = 0.05 means 5% of all transactions contain both A and B. Low support = rare combination — may not be worth acting on. |
| Confidence | How often B appears in transactions that contain A | P(B|A) = transactions with both A and B / transactions with A | Confidence = 0.72 means 72% of customers who buy A also buy B. This is the "reliability" of the rule. |
| Lift | How much more likely B is to be purchased given A — compared to B's baseline frequency | Confidence(A→B) / Support(B) | Lift = 1.0 means no association — buying A tells you nothing about B. Lift > 1 = positive association. Lift = 2.5 means B is 2.5× more likely when A is in the basket. Lift is the most important metric for identifying genuinely useful rules. |
Bread has confidence = 0.85 for the rule "Beer → Bread" (85% of customers who buy beer also buy bread). Sounds impressive. But if 82% of ALL customers buy bread regardless of what else they buy, the rule is almost useless. Lift = 0.85/0.82 = 1.04 — barely better than random. High confidence alone is misleading if the consequent is a very popular item. Always use lift as the primary filter. Lift > 1.5 is typically considered meaningful for action.
| Rule (A → B) | Support | Confidence | Lift | Business Action |
|---|---|---|---|---|
| Nappies → Beer | 3.2% | 68% | 3.1 | Place beer near nappy aisle; bundle promotions |
| Pasta → Pasta Sauce | 8.4% | 81% | 2.8 | Display together; cross-sell at checkout |
| Espresso Pods → Milk | 5.1% | 74% | 2.2 | Bundle in subscription box; cross-discount |
| Bread → Milk | 22% | 87% | 1.1 | High confidence but low lift — both are staples bought by everyone. No targeted action needed. |
The Apriori Algorithm
The Apriori algorithm efficiently generates association rules by first finding all frequent itemsets (those meeting the minimum support threshold), then deriving rules from those itemsets that meet the minimum confidence threshold. It uses the property that any subset of a frequent itemset must also be frequent — dramatically reducing the number of candidate itemsets to evaluate.
Cluster Validation
The elbow told you K. The silhouette tells you if that K is actually good.
Cluster validation answers a fundamental question: are these clusters real, stable, and well-separated — or are they just a mathematical artefact of the algorithm? Three validation approaches work together to answer this.
1. Silhouette Analysis
The silhouette score for each point measures: how compact is its own cluster vs how far away is the nearest other cluster? A score near +1 means the point fits well in its cluster. Near 0 means it is on the boundary. Negative means it probably belongs to a different cluster.
| Average Silhouette Score | Cluster quality | Action |
|---|---|---|
| 0.71 to 1.00 | Strong — clusters are well-separated and compact | Use this K with confidence |
| 0.51 to 0.70 | Reasonable — clusters are distinct but some overlap | Acceptable for most business applications |
| 0.26 to 0.50 | Weak — clusters overlap significantly | Consider a different K or a different algorithm |
| < 0.25 | Poor — no real cluster structure found | The data may not have meaningful groups. Do not act on these clusters. |
2. Stability Testing
Run the same clustering on a random 80% sample of your data, then on a different 80% sample. Do the same clusters emerge? Do the same observations land in the same clusters? Stable clusters replicate; coincidental groupings do not. This is the single most important validation test for business use.
3. External Validation
Test whether the clusters predict outcomes the business cares about. Run an ANOVA: do the clusters significantly differ on churn rate, lifetime value, or complaint frequency? If the clusters do not separate on any meaningful external variable, the segmentation — however internally coherent — may have no business value.
Step 1: Run elbow method to narrow to 2–4 candidate K values. Step 2: Run silhouette analysis on each candidate K — pick the K with the highest average score. Step 3: Test stability by re-running on a holdout sample. Step 4: Profile clusters and test external validation (ANOVA on outcome variables). Step 5: Only then name the clusters and present to stakeholders.
SPSS & Tableau
K-means and hierarchical clustering in SPSS. One-click clustering in Tableau. No code required.
Tableau's built-in clustering applies K-means automatically — the most accessible entry point for non-programmers.
Python — scikit-learn, mlxtend & yellowbrick
Complete segmentation pipeline — from elbow and silhouette through clustering, profiling, and association rules.
Three Simulators
Build intuition for K selection, cluster profiling, and association rules through live interaction.
Simulator 1 · K-Means Elbow Visualiser
A customer dataset is clustered at different values of K. Watch how the scatter plot and WCSS change — and identify where the "elbow" occurs.
Simulator 2 · Cluster Profile Interpreter
Four customer clusters have been identified. Review the profile statistics and assign the most appropriate segment name to each cluster.
Simulator 3 · Association Rules Evaluator
Six association rules are displayed with their support, confidence, and lift scores. Decide which rules are genuinely actionable and which should be filtered out — and why.
Common Mistakes
Segmentation mistakes can lead to strategies built on coincidence rather than real customer differences.
Skipping standardisation before K-means
If annual revenue ($0–$500K) and satisfaction score (1–5) are both used, the revenue variable will dominate entirely. Every cluster difference will reflect revenue variation, not the full variable profile. Always standardise first.
Accepting the first K-means solution without validation
K-means starts randomly. Two runs with the same K can produce very different clusters. Always run with multiple initialisations (n_init=20) and take the best result. Then validate with silhouette scores and stability testing.
Using high confidence as the sole rule filter
High confidence on a popular item (bread, milk) tells you almost nothing. The lift metric controls for base rate popularity. Filter by lift > 1.5 as a minimum — otherwise you will act on rules that are not better than random.
Treating discovered clusters as facts
Clusters are hypotheses. The algorithm always finds K clusters whether they are real or not. Present them as "proposed segments" and validate before building strategy. A cluster solution that fails stability testing should not drive a marketing budget.
Including too many irrelevant variables in clustering
Adding unrelated variables (ID numbers, dates, irrelevant demographics) dilutes the clustering signal. Each variable should measure something conceptually relevant to the segmentation purpose. More is not better.
Not profiling clusters in business terms
Cluster 1 has centroid (Z-score: 1.4, −0.8, 0.3) is meaningless to a marketing director. Always convert back to original units and create a named profile: "Champions: average spend $320/mo, 18 transactions/year, loyal 4+ years." The business value is in the profile, not the algorithm output.