When You Have Too Many Variables — Simplify Without Losing Signal
The final frontier of the analytics ladder. Turn high-dimensional, complex data into structures humans can understand and act on.
Modern data is high-dimensional. A customer survey has 40 questions. A financial model has 80 indicators. A genomics dataset has thousands of gene expression values. Dimensionality reduction compresses this complexity into a smaller number of meaningful dimensions without discarding the essential information.
These techniques serve two distinct purposes: computational (reducing variables before running another model, removing multicollinearity, speeding up algorithms) and interpretive (visualising complex data, discovering hidden structure, making patterns legible to the human eye).
Components
from n dimensions
Principal Component Analysis (PCA)
Transforms correlated variables into uncorrelated principal components ordered by the variance they explain. The workhorse of dimensionality reduction — linear, fast, interpretable.
Factor Analysis (FA)
Identifies latent factors underlying observed variables. Unlike PCA (variance-focused), FA models shared variance and is explicitly theory-driven. Covered here as a dimensionality reduction complement to PCA.
t-SNE
t-Distributed Stochastic Neighbour Embedding. Non-linear technique for 2D/3D visualisation. Preserves local structure — nearby points in high-dimensional space cluster together in the plot. Excellent for exploratory analysis and cluster discovery.
UMAP
Uniform Manifold Approximation and Projection. Faster than t-SNE, better at preserving global structure. Increasingly preferred for large-scale exploratory visualisation in genomics, customer analytics, and text analysis.
Canonical Correlation Analysis (CCA)
Examines the relationship between two sets of variables simultaneously. Finds linear combinations of each set that are maximally correlated. The multivariate extension of simple correlation.
Multidimensional Scaling (MDS)
Visualises the similarity or dissimilarity between objects in a low-dimensional map. Starts from a distance matrix (not raw data) and arranges items so that similar items are close together in 2D.
Autoencoders (intro)
Neural network architecture for non-linear dimensionality reduction. An encoder compresses the input to a low-dimensional "bottleneck"; a decoder reconstructs the original. The bottleneck representation is the reduced-dimension encoding.
PCA: Use when the goal is data compression, removing multicollinearity before regression, or preparing data for machine learning. No underlying theory required — just variance maximisation. Factor Analysis: Use when you have a theoretical model of latent constructs (e.g. "intelligence," "brand equity," "service quality") and want to test whether observed variables reflect those constructs. Theory-driven, confirmatory or exploratory.
Principal Component Analysis — Foundations
PCA finds the directions of maximum variance in high-dimensional data and projects everything onto those directions.
Imagine 50 customer satisfaction questions, most of which are correlated with each other. PCA finds a small number of principal components — new variables that are linear combinations of the original questions — that capture most of the variation in the data. The first principal component captures the most variance; the second captures the most of what remains; and so on.
The Three Core Concepts
Variance Explained
Each principal component explains a percentage of the total variance in the data. The first component typically explains the most. A good PCA retains components that together explain 70–80% of total variance.
Reported as a "scree plot" (eigenvalue vs component number) and a "cumulative variance explained" table.
Loadings
Each component has a loading for each original variable — showing how strongly that variable contributes to the component. High positive loading = the variable rises when the component rises. High negative loading = inverse relationship.
Use loadings to name components: a component with high loadings on "queue time," "response speed," and "resolution time" is a "service speed" dimension.
Scores
Each observation (customer, product, period) receives a score on each component. Scores are the new reduced-dimension representation of the data — they replace the original 50 variables with 3–5 component scores.
Use scores in subsequent analysis: cluster analysis, regression, or visualisation — each customer now has 3 coordinates instead of 50.
Orthogonality
All principal components are uncorrelated with each other (orthogonal). This is a critical advantage: it eliminates multicollinearity when using PCA scores as predictors in regression.
This is the fundamental difference from the original variables, which were correlated. PCA transforms correlated inputs into uncorrelated components.
How many components to retain?
| Rule | Criterion | How to apply | Limitation |
|---|---|---|---|
| Kaiser Criterion | Retain components with eigenvalue > 1.0 | Each component should explain more variance than a single original variable (eigenvalue = 1.0 is the baseline) | Can retain too many components with large datasets |
| Scree Plot Elbow | Retain components before the "elbow" bend | Plot eigenvalues; retain components before the line flattens sharply | Elbow is sometimes ambiguous or gradual |
| Cumulative Variance | Retain enough components to explain 70–80% of variance | Sum eigenvalues as percentages; stop when cumulative % exceeds threshold | Threshold is arbitrary — 70% may discard important detail |
| Interpretability | Retain components that can be meaningfully named | A component whose loadings form no interpretable pattern is noise — drop it | Subjective — but essential for business communication |
PCA maximises variance. If variables are in different units (revenue in $000s vs satisfaction on 1–10), the high-variance variable (revenue) will dominate the first component entirely — regardless of its analytical importance. Always standardise all variables to mean=0, SD=1 (Z-scores) before running PCA. In SPSS: tick "Correlation matrix" in the Factor Analysis options. In Python: before .
PCA — Reading & Interpreting Output
The maths runs itself. The analytical skill is in reading the output and translating it into business meaning.
A company surveys 500 customers on 10 aspects of their experience. PCA reveals 3 components explaining 74% of total variance:
| Survey Item | PC1 Loading | PC2 Loading | PC3 Loading |
|---|---|---|---|
| Queue wait time | −0.82 | 0.12 | 0.08 |
| Resolution speed | −0.79 | 0.08 | 0.15 |
| First-call resolution | −0.75 | 0.18 | 0.11 |
| Staff friendliness | 0.14 | 0.81 | 0.09 |
| Staff knowledge | 0.09 | 0.76 | 0.18 |
| Problem understanding | 0.21 | 0.72 | 0.15 |
| Website ease of use | 0.08 | 0.11 | 0.85 |
| App responsiveness | 0.11 | 0.09 | 0.80 |
| Digital self-service | 0.15 | 0.14 | 0.77 |
| Overall NPS | −0.41 | 0.44 | 0.38 |
| % Variance | 31% | 26% | 17% |
PC1: High negative loadings on wait time, resolution speed, first-call resolution → label: "Service Efficiency" (lower scores = faster, better service).
PC2: High positive loadings on friendliness, knowledge, problem understanding → label: "Staff Quality".
PC3: High positive loadings on website, app, digital self-service → label: "Digital Experience".
The 10-variable survey is now described by 3 actionable dimensions: Efficiency, Staff Quality, Digital. Each customer has a score on each — enabling targeted interventions by segment.
Using PCA scores in downstream analysis
| Next step | How PCA scores help | Advantage over raw variables |
|---|---|---|
| Regression | Use PC scores as predictors. E.g. predict NPS from Efficiency, Staff Quality, Digital scores. | Eliminates multicollinearity among predictors. Coefficients are stable and interpretable. |
| Clustering | Run K-means on 3 PC scores instead of 10 raw variables. Faster and avoids scale issues. | Fewer dimensions means less noise; clustering is more stable. |
| Visualisation | Plot PC1 vs PC2 as a 2D scatter plot. Each customer is a point — spatial proximity = similar experience. | Makes 10-dimensional data visually legible. Cluster structure and outliers become visible. |
| Anomaly detection | Observations with extreme scores on any component are outliers relative to the component's dimension. | Detects unusual customers, fraudulent patterns, or measurement errors efficiently. |
Factor Analysis as Dimensionality Reduction
While PCA extracts variance, Factor Analysis extracts meaning — the latent constructs that explain why variables correlate.
Set 05 introduced EFA and CFA in the context of testing theoretical models. Here we revisit Factor Analysis specifically as a dimensionality reduction tool — its role in compressing many observed variables into a smaller number of interpretable constructs before further analysis.
PCA vs Factor Analysis — a clear comparison
| Dimension | PCA | Factor Analysis |
|---|---|---|
| Goal | Maximise variance explained in the data | Explain shared variance — the co-variation due to latent factors |
| Theory needed? | No — purely data-driven | Optional for EFA; required for CFA |
| Components/Factors | Ordered by variance explained (PC1 captures most) | Factors represent underlying constructs — order is not inherent |
| Uniqueness | All variance is accounted for by components | Each variable has a unique variance component (error) not attributed to any factor |
| Rotation | Optional (orthogonal/oblique) | Standard — Varimax (orthogonal) or Oblimin (oblique) improves interpretability |
| Output used for | Data compression, feature engineering for ML, removing multicollinearity | Scale construction, survey validation, theoretical construct testing |
| Primary tool | SPSS, Python (sklearn.decomposition.PCA) | SPSS, IBM AMOS, Python (factor_analyzer) |
Choose FA when: (1) Your variables are survey items or scale measures designed to reflect underlying constructs. (2) You need to validate construct validity (Cronbach's alpha, AVE, CR). (3) You plan to use the factors in SEM (Set 05). (4) You want rotation to improve interpretability — FA rotations are theoretically motivated, PCA rotations are not. Choose PCA when you simply need to reduce dimensionality without a theoretical model.
An HR team administers a 20-item engagement survey. EFA with Oblimin rotation identifies 4 factors: Autonomy (items 1–5), Relationship Quality (items 6–10), Growth Opportunities (items 11–15), Recognition (items 16–20). Each factor's items are averaged to create a factor score. These 4 scores replace the 20 raw items in subsequent regression predicting turnover intention. Result: a clean, interpretable model where each factor's unique contribution to turnover can be estimated — with no multicollinearity.
t-SNE & UMAP
Non-linear dimensionality reduction for visualisation. See the structure in high-dimensional data that PCA cannot reveal.
PCA is linear — it only finds straight-line directions of maximum variance. Many real datasets have complex, curved, non-linear structures that PCA misses entirely. t-SNE and UMAP are non-linear techniques specifically designed to produce 2D or 3D visualisations where meaningful clusters and structures in high-dimensional data become visible.
t-SNE — t-Distributed Stochastic Neighbour Embedding
t-SNE converts high-dimensional distances between points into probabilities, then finds a 2D arrangement of points that best preserves those probability relationships. Points that were nearby in high-dimensional space end up clustered together in the 2D plot.
| Property | Detail |
|---|---|
| Perplexity | The most important parameter. Controls the "neighbourhood size" — how many nearby points t-SNE considers when calculating probabilities. Typical range: 5–50. Lower = focuses on very local structure. Higher = considers broader neighbourhood. Try multiple values and compare. |
| Local vs Global structure | t-SNE preserves local structure (nearby points cluster correctly) but does NOT preserve global structure (the distance between clusters on the plot is not meaningful). Never interpret the relative distances between clusters in a t-SNE plot. |
| Stochastic | Results vary between runs (random initialisation). Set a random seed for reproducibility. Run multiple times to confirm that cluster patterns are stable. |
| Scale | Slow on large datasets (>50,000 points). Use approximate t-SNE (BH-t-SNE) or switch to UMAP for large data. |
UMAP — Uniform Manifold Approximation and Projection
UMAP is a newer technique that addresses t-SNE's limitations while producing equally compelling visualisations. It is faster, scales better to large datasets, preserves more global structure, and can be used as a general-purpose dimensionality reduction tool (not just for visualisation).
Speed
UMAP runs 10–100× faster than t-SNE on large datasets. Practical for exploratory analysis of millions of records in minutes.
Global structure preserved
Unlike t-SNE, the relative positions of clusters in a UMAP plot are more meaningful — similar clusters in high-dimensional space appear closer in the 2D plot.
Deterministic option
UMAP can be set to produce the same result every run, making it easier to incorporate into reproducible analysis pipelines.
n_neighbors
Controls the balance between local and global structure. Small values (5–15) emphasise fine local detail. Larger values (30–100) emphasise broad global patterns.
Both are for exploration only. The 2D plots cannot be used for quantitative analysis, prediction, or statistical testing. They show that clusters exist — not why. The axes have no meaningful units. Cluster sizes in the plot are artefacts of the algorithm, not reflections of actual group sizes. Use these for hypothesis generation and visual communication — not for drawing analytical conclusions.
A telecom company has 80,000 customers described by 50 behavioural variables (call patterns, data usage, tenure, payment history, etc.). PCA reduces to 10 components (explaining 72% of variance). UMAP then projects those 10 components to 2D. The 2D plot reveals 5 distinct customer clusters — which are then profiled (Set 07 methods) and used to design targeted retention programmes. The entire pipeline: standardise → PCA → UMAP → cluster → profile → act.
CCA, MDS & Autoencoders
Three more advanced techniques for specific multivariate reduction problems.
Canonical Correlation Analysis (CCA)
CCA finds the relationship between two sets of variables simultaneously — the multivariate extension of simple correlation. It identifies linear combinations of Set 1 variables and Set 2 variables that are maximally correlated with each other.
Set 1 (Employee): Autonomy score, Manager rating, Workload satisfaction, Growth perception (4 variables).
Set 2 (Customer): NPS, Satisfaction score, Repeat purchase rate, Complaint rate (4 variables).
CCA finds the combination of employee experience variables that best predicts the combination of customer outcome variables. First canonical variate: Autonomy + Manager rating → NPS + Satisfaction (r = 0.68). Interpretation: employees who feel autonomous and well-managed produce customers who score higher on NPS and satisfaction simultaneously.
Multidimensional Scaling (MDS)
MDS takes a distance or dissimilarity matrix (not raw data) and produces a 2D or 3D map where items that are most similar are placed closest together. It reveals the perceptual or conceptual structure underlying a set of pairwise comparisons.
Customers rate the similarity of 8 competing brands on a 1–10 scale (how similar is Brand A to Brand B?). MDS converts these similarity ratings into a 2D perceptual map. The two axes that emerge (after interpretation) represent: X-axis = "Value vs Premium" · Y-axis = "Traditional vs Innovative." Brands cluster into quadrants: premium-traditional (Brands A, B), value-innovative (Brands E, F), etc. The map shows where white space exists for product positioning and reveals which brands are direct competitors.
Autoencoders — Non-linear Dimensionality Reduction
An autoencoder is a neural network trained to compress input data into a lower-dimensional "bottleneck" layer and then reconstruct the original data from that compressed representation. The bottleneck activations are the reduced-dimension features.
| Component | Role | Business meaning |
|---|---|---|
| Encoder | Compresses input (e.g. 100 variables) to bottleneck (e.g. 5 dimensions) | Learns the most efficient compressed representation of the data |
| Bottleneck layer | The reduced-dimension representation | These 5 values replace the original 100 — the learned features |
| Decoder | Reconstructs the original data from the bottleneck | Reconstruction quality validates the encoding — poor reconstruction = information lost |
| Reconstruction error | Difference between input and decoded output | High error for a specific observation = anomaly (used in fraud detection) |
PCA is linear — it can only find straight-line compressions. Autoencoders are non-linear (due to activation functions) — they can find complex curved compressions that better capture non-linear structure. Rule of thumb: try PCA first (faster, interpretable). If PCA leaves too much unexplained, try an autoencoder. In Python: or . For anomaly detection specifically, autoencoders are the preferred approach — observations with high reconstruction error are flagged as anomalous.
SPSS & Excel
PCA and Factor Analysis in SPSS. Manual PCA walkthrough in Excel.
SPSS runs PCA and Factor Analysis through the same menu — the distinction is made in the extraction method settings.
• For PCA: Method = Principal Components
• For FA: Method = Principal Axis Factoring or Maximum Likelihood
Analyse: Correlation matrix (this standardises variables automatically)
Extract: Eigenvalues over 1 (Kaiser criterion) or specify number of factors
Tick Unrotated factor solution and Scree plot → Continue
• Orthogonal (uncorrelated factors): Varimax
• Oblique (correlated factors, more realistic): Direct Oblimin
Tick Rotated solution and Loading plot → Continue
Click Options → Suppress small coefficients: 0.40 → Continue → OK
Excel does not have a built-in PCA function, but the Data Analysis ToolPak provides the components needed to do it manually — useful for understanding the mechanics.
Python — sklearn, umap-learn & more
Complete dimensionality reduction pipeline — from PCA through t-SNE, UMAP, and autoencoder basics.
Three Simulators
Build intuition for PCA, component interpretation, and the effect of dimensionality on data structure.
Simulator 1 · PCA Variance Explorer
A 10-variable dataset is analysed by PCA. The eigenvalues are shown. Adjust the cumulative variance threshold and see how many components you need to retain — then evaluate the trade-off.
Simulator 2 · PCA Loading Pattern Interpreter
Three PCA loading tables are shown. For each, interpret the loading pattern and name the component based on which variables load highly.
Simulator 3 · Technique Selector
Six analytical situations are described. Match each to the correct dimensionality reduction technique.
Common Mistakes
Dimensionality reduction mistakes are particularly hard to detect — the output always looks plausible.
Not standardising before PCA
A variable with large numerical values will dominate the first component regardless of its analytical importance. Annual revenue in $000s will overwhelm a satisfaction scale (1–10). Always Z-score all variables first. In SPSS: use Correlation Matrix extraction. In Python: before .
Interpreting t-SNE/UMAP inter-cluster distances
The distance between clusters in a t-SNE or UMAP plot is not meaningful. Two clusters that appear far apart may not be more dissimilar than two clusters appearing close. These plots reveal cluster existence and membership — not quantitative inter-cluster relationships.
Retaining too few components without checking
Applying Kaiser's rule rigidly (eigenvalue > 1) may keep too few or too many components. Always cross-check with the scree plot and with interpretability. A component that cannot be meaningfully named is worth questioning — even if its eigenvalue exceeds 1.
Confusing PCA components with factor analysis factors
Components (PCA) are mathematical constructions — they explain variance and may span all original variables. Factors (FA) are theoretical constructs — they model shared variance and each variable should load cleanly on one factor. The same label is sometimes misapplied to both, but they have different assumptions, outputs, and uses.
Using PCA without validating that the data is suitable
Run the KMO (Kaiser-Meyer-Olkin) test and Bartlett's test of sphericity before PCA/FA. KMO < 0.6 suggests the variables are not sufficiently correlated for factor analysis — PCA would extract mostly noise. Bartlett's p > 0.05 means the correlation matrix is essentially an identity matrix — no structure to extract.
Applying t-SNE to small datasets or for quantitative analysis
t-SNE needs at minimum several hundred observations to produce stable, meaningful layouts. On small datasets (n < 200), results are unreliable. More importantly: t-SNE output cannot be used for statistical testing, regression, or any quantitative analysis. It is a visualisation tool — not an analytical one.