The Discipline of Analytical Rigor
Three cards on what separates work that survives challenge from work that only looks like it would.
Modules 01–03 built the mindset, the structure, and the focus. Module 04 is where modelling tools finally come into full play — and where the discipline tightens around them. The legitimacy ladder has been climbing for three modules; Card F opened the door to inferential methods, and Cards G–I narrowed the set of hypotheses worth modelling. Module 04 is the work of actually doing the modelling — and of stress-testing what it produces.
The three cards form a tight sequence. Card J is the cheap pre-modelling check: before any model is built, does the explanation make sense at the order of magnitude required? Card K is what happens once a model is built: it does not reveal truth — it formalises a set of assumptions, and its outputs are conditional on those assumptions. Card L is what happens after a result emerges: the work is not done when a finding appears, but when the finding has been actively challenged and either survived or been revised.
Fast-Track Validation
Sanity-checking logic before building models. Why a 15-minute plausibility check is more rigorous than a multi-week regression on the same hypothesis.
Model Choice & Assumptions
What models really say — and what they don't. Why choosing a model is choosing a belief structure, and what each common method is quietly assuming.
Learning by Disproving
Robustness, sensitivity, analytical humility. Why the initial result is the starting point of the work, not its end — and what selective reporting hides.
All three cards share an orientation: strong analysis actively tries to disprove itself. Card J disproves before modelling (does the math even allow this?). Card K disproves the model's claim to objectivity (which assumptions would have to hold?). Card L disproves the result itself (does the conclusion survive alternative specifications?). This is the same falsification discipline introduced in Card G, applied at three different stages of the analytical workflow.
The Rush to Model
Validation comes before verification. Skipping the order is one of the most costly errors in applied analytics.
This card addresses a frequent and costly analytical failure: jumping into formal modelling before basic logic has been tested. In many organisations, the moment a hypothesis is formed, the next step is to open a notebook, query a database, and start building. Regression models are run. Machine-learning pipelines are stood up. Dashboards are constructed. The work feels rigorous because tools are being used. But often, the underlying logic of the explanation has never been examined for plausibility.
This rush to model is not laziness or carelessness. It is the natural consequence of a culture that equates sophistication with rigor. Spreadsheet calculations look like back-of-the-envelope work; regression models look like analysis. The analyst who runs a quick mental calculation and rejects a hypothesis on plausibility grounds may feel they have not done enough. The analyst who builds a complex model on the same hypothesis appears thorough — even if the hypothesis was implausible from the start.
A regression model run on an implausible hypothesis will return coefficients with confidence intervals. The output will look definitive. But the precision is misleading: the model has fit noise to a story that could not be true even if every assumption held. The analyst now possesses confident-looking results for a claim that should never have been investigated. Worse, those results are difficult to challenge — the math is correct, the methodology is standard, and the only flaw is one no statistical diagnostic will catch: the hypothesis was implausible to begin with.
Validation vs verification
A foundational distinction at this stage is the difference between validation and verification. The two words sound similar and are often used interchangeably in casual conversation. In disciplined analytical work, they refer to very different activities, and confusing them is one of the most common reasons flawed analysis passes review.
| Activity | What it asks | When it should happen |
|---|---|---|
| Validation | Does the underlying logic of this explanation make sense? Could this hypothesis plausibly be true given what we already know about the system, the magnitudes involved, and the constraints? | First. Before any model is built. Validation is the gate that determines whether a hypothesis is worth investigating with formal tools. |
| Verification | Does this specific model produce statistically acceptable results? Are the coefficients significant? Does the model fit the data well? Are the diagnostics clean? | Second. After validation has confirmed the logic is plausible. Verification answers a different and narrower question: is the implementation correct? |
Validation must always come first. A model that passes verification (good fit, significant coefficients, clean diagnostics) but fails validation (the logic is implausible) produces confident wrong answers. A model that fails verification but would have passed validation can be repaired. The order matters because flawed logic cannot be fixed downstream. Once an implausible hypothesis is dressed in statistical machinery, the implausibility becomes invisible.
Advanced analytical thinking requires a deliberate pause between hypothesis formulation and formal modelling. At this stage, the analyst's goal is not precision, optimisation, or prediction. The goal is to answer a simpler but more important question: does this explanation make sense at all?
The Plausibility Check
15 minutes of arithmetic that can prevent weeks of confidently wrong analysis.
The plausibility check is the analytical move that distinguishes mature practitioners from novices. To make it concrete, consider an example. An analyst at a consumer-goods company is asked to investigate a 6% decline in quarterly revenue. A stakeholder proposes a hypothesis: "The decline is being driven by the new packaging redesign that launched in week three of the quarter. Customers are confused by the new look and not finding the product on shelves."
Before building any model, the disciplined analyst runs a quick plausibility check:
2. The product accounts for ~40% of total revenue
3. Even if every customer was affected (extreme upper bound), max impact = 40% × 77% × effect size
4. To produce 6% revenue decline → packaging would have to depress this product by ~19.5%
5. Historical benchmark on packaging changes: 1–3% typical effect
→ Conclusion: The hypothesis cannot explain the observed magnitude.
This entire check takes about 15 minutes. It does not produce a precise answer; it produces a precise rejection. The packaging hypothesis cannot be the primary cause of the observed decline — the math does not support it. The analyst can now redirect attention to other candidate drivers without having spent weeks building a model on a hypothesis that was never plausible.
The check did not prove the packaging redesign had no effect; it almost certainly had some effect. What the check established is that the packaging redesign cannot be the dominant cause of the 6% decline. Other drivers must be at work. The analyst's next step is not to model packaging more carefully — it is to look elsewhere.
If the analyst had skipped the plausibility check and gone directly to modelling, several weeks would have been spent building a sophisticated regression of revenue against packaging-related variables. The model would have produced statistically significant coefficients. The output would have been presented to leadership. A multi-million-dollar packaging-revision project might have been launched to "fix" the issue.
Six months later, when revenue continued to decline, the actual cause would surface — because it had been there all along, just unexamined. The plausibility check was the cheap insurance that prevented an expensive mistake.
Practice · Plausibility Checker
Each scenario presents a stakeholder hypothesis and an observed effect. For each input, pick the value that the math actually allows. Then give your verdict: plausible, implausible, or inconclusive. Per-step feedback reveals the disciplined reasoning.
Simple Tools, Used Well
The willingness to use the simplest tool that can answer the question is the mark of analytical maturity.
At the plausibility stage, simple analytical tools are not only sufficient — they are preferred. The discipline being taught is the willingness to use the simplest tool that can answer the question being asked. For plausibility checks, that tool is almost always a spreadsheet, a calculator, or a sketched calculation.
| Tool | What it does | When to reach for it |
|---|---|---|
| Back-of-the-envelope calculations | Quick arithmetic that establishes whether a proposed effect could plausibly produce the observed outcome. Order-of-magnitude reasoning, not precision. | Always, before any formal model. The default first move on any new hypothesis. |
| Rough scenario tables | Simple matrices that examine outcomes under best-case, worst-case, and base-case assumptions for the main parameters. | Testing whether a hypothesis is consistent with observed magnitudes under different parameter assumptions. |
| Simple ratios & rates | Per-customer, per-period, per-unit comparisons that ground the analysis in interpretable benchmarks. | Sanity-checking whether the implied behaviour is reasonable at the unit level. |
| Spreadsheet logic checks | Light models that test whether the proposed mechanism produces the observed effect when reasonable inputs are applied. Excel as a thinking engine, not a reporting tool. | When a single calculation is too coarse but a full model is still premature. |
Formal statistical models, optimisation routines, and machine-learning techniques are not yet legitimate at the plausibility stage. Using them before logic has been validated risks investing weeks of effort in ideas that should have been discarded in 30 minutes. The principle is not that sophisticated tools are bad; it is that they are expensive — in time, attention, and credibility — and they should be reserved for hypotheses that have already passed plausibility.
Why this runs against the cultural pull
This card reinforces a professional norm that runs against the cultural pull toward sophistication: analytical rigor begins with logical plausibility, not with sophisticated tools. The analyst who rejects a weak hypothesis with a 15-minute calculation has done more rigorous work than the analyst who builds a regression model on the same weak hypothesis. The first protected the organisation from a confident wrong answer. The second produced one.
In many organisations, simple calculations are treated as inferior to complex models. Decks full of regression output look like serious analysis; a spreadsheet with a few ratios looks like an undergraduate exercise. The discipline is to recognise that these appearances are misleading. The analytical value of a tool is determined by how well it answers the question being asked, not by how impressive it looks in the room.
Models Don't Reveal Truth
They formalise assumptions — and then estimate parameters inside the framework those assumptions create.
This card addresses one of the most dangerous misconceptions in analytical work: the belief that models produce truth. The intuition is widespread, the language reinforces it ("the model says," "the data shows," "the algorithm found"), and the visual outputs of modelling tools encourage it — precise coefficients, confidence intervals, performance metrics that look definitive. The result is a culture in which model outputs are often treated as objective findings rather than as conditional inferences.
In practice, models do not reveal reality. They encode assumptions about how reality works — and then estimate parameters within the framework those assumptions create. Linear regression assumes that effects add up in straight lines. Logistic regression assumes that probabilities behave a certain way. Tree-based methods assume that the data can be carved into rectangular regions. Neural networks assume that arbitrary functions can be approximated through layered transformations. None of these are facts about the world. They are choices about how to represent the world.
When analysts misunderstand or ignore these assumptions, model outputs can appear authoritative while being fundamentally misleading. A linear regression will dutifully estimate coefficients even when the underlying relationship is highly nonlinear. The math will be correct. The output will look standard. But the answer will misrepresent the reality the analyst is trying to understand. The model has not lied; it has done exactly what it was asked to do under assumptions that did not hold.
The pull toward familiar models
As analytical tools become more accessible, the temptation to select models based on availability, familiarity, or perceived sophistication grows stronger. Linear regression is often applied by default because it is well known. Logistic regression is reached for whenever an outcome is binary. Machine-learning models are deployed because they can be deployed. The deeper question — whether the model's assumptions match the problem being analysed — is often skipped entirely.
A common belief is that machine-learning models, ensemble methods, or neural networks somehow "let the data speak for itself" and avoid the assumption-laden problems of classical regression. This is incorrect. More complex models do not eliminate assumptions; they merely hide them.
A random forest assumes the data can be partitioned by axis-aligned splits. A gradient-boosted model assumes errors can be successively corrected by additive weak learners. A neural network assumes the function being approximated lies within the class representable by its architecture. These assumptions are less visible than the assumptions of linear regression — which makes them more dangerous, not less. The disciplined analyst asks the same question regardless of model sophistication: what is this method assuming, and would those assumptions hold for my problem?
A worked example: when linearity fails
An analyst models how customer demand responds to price using linear regression. The model estimates that a $1 price increase reduces demand by 4 units. The coefficient is significant. The R² is acceptable. The output looks clean.
In reality, demand is highly nonlinear: at low prices, customers are insensitive; at moderate prices, sensitivity grows; at high prices, demand collapses entirely as customers switch to substitutes. The linear model has averaged across these regimes and reported a single number that is approximately right on average and substantially wrong at every actual price point.
Recommendations based on the model — such as "we can raise prices by $3 with limited demand impact" — may be dangerously incorrect because the linear assumption does not hold over the relevant range. The math was right. The choice of model was wrong.
Choosing a model is not a technical step — it is a conceptual commitment. The analyst who selects a model is making a claim about how the underlying system behaves, whether they realise it or not. Doing this consciously, with awareness of the assumptions being made and the consequences of being wrong about them, is the discipline this card establishes.
What Common Models Assume
Match the model to the question — not the question to the model.
To select a model responsibly, the analyst must understand what each model is assuming about the system being studied. The assumptions are rarely explicit in the modelling tool itself; they live in the mathematical structure of the model and become visible only when the analyst examines them deliberately.
| If the question is about… | Outcome type | Appropriate model family | Key assumptions |
|---|---|---|---|
| How much (continuous magnitude) | Continuous variable (revenue, cost, time) | Linear regression, generalised linear models | Linearity of relationships, additivity, stability across the predictor range |
| Whether (binary classification) | Yes/no, will/won't, has/hasn't | Logistic regression, classification trees | Probabilities are the appropriate target; predictors influence log-odds linearly |
| When (timing of event) | Time-to-event with possible censoring | Survival analysis (Cox models, Kaplan-Meier) | Hazard rates can be modelled; censoring is non-informative |
| Which group (multi-class) | Categorical with multiple classes | Multinomial regression, classification methods | Categories are mutually exclusive; a probabilistic structure is appropriate |
This mapping is not a universal rule — there are situations where less common models are appropriate, and many real questions involve combinations of these. The point is that the model is selected because it matches the question, not because it is convenient. When the analyst finds themselves reshaping the question to fit the model ("let's just dichotomise the outcome" or "let's just assume linearity for now"), that is a signal to step back and reconsider whether a different model family would serve the question better.
The danger signal: reshaping the question to fit the tool
"Let's just dichotomise the outcome"
Forcing a continuous outcome into a binary one because the analyst is more comfortable with logistic regression. The "how much" question gets answered as a "whether" question — and the magnitude information that the stakeholder actually needed is destroyed in the conversion.
"Let's just assume linearity for now"
Reaching for linear regression when the underlying relationship is plainly nonlinear (saturating returns, threshold effects, regime changes). The model averages across distinct regimes and reports a single coefficient that is approximately right on average and substantially wrong at every actual operating point.
"The XGBoost result must be right — the F1 score is great"
Trusting a complex model precisely because its assumptions are less visible. The model fit a complex pattern but with assumptions that may not generalise. Hidden assumptions are still assumptions — they just become harder to surface, audit, or stress-test.
Practice · Model Match
Each scenario presents a business question. Pick (a) the appropriate model family, (b) the most consequential assumption that family embeds, and (c) what would invalidate that assumption. Per-pick feedback explains the verdicts.
Reading Model Outputs Conditionally
A coefficient is not a finding by itself — it is a finding under a model's assumptions.
Once the analyst understands what their model is assuming, the way they read its output must change. Model diagnostics, coefficients, and performance metrics must always be interpreted in light of assumptions, not in isolation. A coefficient of 0.34 with p<0.001 is not a finding by itself. It is a finding under the model's assumptions. If those assumptions hold, the coefficient is meaningful. If they do not, the coefficient is an artefact of the model rather than a fact about the world.
Two ways to communicate the same result
| Unconditional language | Conditional language |
|---|---|
| "The model proves that pricing drives churn at a rate of 4.2% per dollar increase." | "Under the assumptions of the linear model, the data is consistent with a churn effect of 4.2% per dollar of price increase, within the observed price range. If the relationship saturates above $X, the estimate would shift downward." |
| "The data shows that customer-success calls drive renewal." | "Holding the observable controls fixed, accounts receiving more CSM calls renewed at a higher rate. Whether this reflects a causal effect depends on whether the CSM team's call selection is independent of unobserved retention factors." |
| "The algorithm found three segments." | "Under the clustering algorithm's distance metric and chosen number of clusters, the data partitions into three groups. A different metric or cluster count would produce a different partition." |
When stakeholders later discover that conclusions depended on assumptions the analyst did not realise they were making, trust in the analytical function erodes. The disciplined alternative is to communicate findings as conditional: "Under the assumptions of the linear model, the effect is estimated at X. Those assumptions are most likely to fail if Y, in which case the estimate would shift toward Z." This honesty is harder to deliver and easier to defend.
Tool legitimacy — fully open, but conditional on all the prior gates
At Card K, formal modelling tools become legitimate — but only because prior cards have disciplined judgment, structure, hypotheses, prioritisation, and logic validation. The analyst arrives at Card K having framed the problem (Card B), structured it into MECE drivers (Card E), defined variables and hypotheses (Card F), prioritised the critical path (Card I), and validated logic with plausibility checks (Card J). Now, finally, formal models are appropriate. The legitimacy is conditional on all of that prior discipline.
Regression models, classification techniques, and other formal methods are appropriate when:
- Variables are well defined — each input has been specified with units, sources, time frame, and boundary (Card F).
- Hypotheses are explicit — the analyst can state, before running the model, what relationships are being tested and what would constitute disconfirming evidence (Card G).
- The analytical question matches the model's assumptions — the mapping introduced in this card is honoured.
- Logic has passed plausibility — the hypothesis being modelled is mathematically capable of producing the observed effects (Card J).
The First Result Is Not the Conclusion
Strong analysis treats initial results not as findings, but as claims that must survive challenge.
This card addresses a common but subtle analytical failure: stopping analysis as soon as a model produces a "good" result. In many professional settings, once a model appears statistically significant, fits the data well, or performs well on a chosen metric, the analytical work is considered complete. The output is presented to stakeholders as a finding. The decision is made. The team moves on. This pattern feels efficient, and it produces results that look rigorous. But it creates a dangerous illusion of certainty — one that often dissolves when the conclusions encounter the messy reality they were supposed to inform.
The trap is structural. Modelling tools produce outputs whether or not the underlying analysis is sound. A regression with significant coefficients looks the same as a regression with significant coefficients that would disappear under a slightly different specification. A model with strong predictive performance looks the same as a model that performs strongly only on the specific data it was trained on. The visible output is identical; the underlying robustness is not.
Three concepts that look similar — and are different
Three concepts must be distinguished at this stage. Each plays a different role in the validation of analytical work, and conflating them is one of the most common reasons fragile findings are presented with unwarranted confidence.
| Concept | What it asks | What it can't tell you |
|---|---|---|
| Confirmation | Does the data support this hypothesis under one specific setup — one model spec, one time window, one sample, one set of assumptions? | Whether the support would persist if the setup changed. Confirmation establishes that a result is possible, not that it is robust. |
| Robustness | Does the support persist when reasonable alternatives are considered? Do different specifications, samples, or time windows produce broadly similar conclusions? | Whether the result is causally correct. Robustness is about stability across analytical choices, not about ruling out confounders. |
| Sensitivity analysis | How much do conclusions change when key assumptions, parameters, or data inputs vary? Which inputs are the conclusions most dependent on? | Whether the variations tested cover the full space of plausible alternatives. Sensitivity analysis only tests what the analyst chooses to vary. |
Confirmation comes first. The analyst produces an initial result that is consistent with the hypothesis. Robustness asks whether that confirmation generalises. Sensitivity analysis examines which inputs the conclusion depends on most. The three together form a layered defence against fragility. Skipping any layer leaves the analysis exposed.
The willingness to attack one's own results often feels uncomfortable — like betraying the work already done. Card L reframes it: a result that has survived challenge is far more defensible than one that has not. The analyst who openly stress-tests their own findings demonstrates more rigor, not less, than the analyst who presents a single confident result without examination. Stakeholders eventually learn the difference.
Model Dependence
When one specification carries the result, the analyst's conclusion is about the modelling decision — not about the world.
A particular form of fragility deserves explicit attention: model dependence. When conclusions rely heavily on one specific modelling choice — a particular regression specification, a particular time window, a particular variable definition, a particular handling of missing data — the analysis becomes vulnerable in a specific way. The result holds, but only because of the choice. A reasonable alternative choice would produce a different result. The analyst's conclusion is not really about the world; it is about the modelling decision they happened to make.
Model dependence is often invisible to the analyst who created the model. They made the choices that produced the result, and the result feels stable because it has not been challenged. But challenge is exactly what reveals model dependence. Trying a different functional form, a different sample, a different lag structure — any of these can expose whether the conclusion was a feature of the world or a feature of the analysis.
An analyst builds a regression that finds a strong, statistically significant relationship between marketing spend and sales. The coefficient is precise, the p-value is small, and the model fits well. The team prepares to recommend a 30% increase in marketing spend.
Before doing so, the analyst tries an alternative: defining marketing spend by month rather than by week, and including a control for the quarterly seasonality cycle. The relationship vanishes. The original finding was an artefact of the time aggregation and missing seasonal control. The result was real under one specification and absent under another reasonable one — a textbook case of model dependence. The discipline of trying alternatives caught the fragility before the recommendation went out.
Advanced analysts recognise that no single model deserves unconditional trust. The remedy is not to use more sophisticated models; it is to use multiple specifications and check whether they agree. When several reasonable approaches converge on the same conclusion, the analyst can speak with confidence. When they diverge, the analyst must report that divergence honestly — not bury it in favour of the specification that produced the most preferred answer.
The stress-testing toolkit
| Tool | What it does |
|---|---|
| Alternative model specifications | Re-run the analysis with different functional forms, different control sets, or different model families. If the conclusion is robust, it survives reasonable alternatives. |
| Variable transformations | Test whether log-transforming variables, using percentages instead of levels, or different aggregations changes the conclusion. Many findings appear or disappear depending on these choices. |
| Different time windows or samples | Examine whether the result holds when the analysis is restricted to a subperiod, a different subset of customers, or a different region. A finding that holds only in one slice of the data is by definition fragile. |
| Sensitivity tables & scenario checks | Examine how outputs change as key parameters vary across plausible ranges. The output is not a single number but a sensitivity surface. |
| Regression diagnostics & assumption toggles | Formal checks of model assumptions, with deliberate exploration of what changes when assumptions are relaxed or modified. |
The goal of all these tools is not to optimise results — it is to stress-test conclusions. The analyst is not looking for the specification that produces the most favourable answer; they are looking for the specifications under which the answer changes. A result that survives many such tests is genuinely strong. A result that breaks under several is fragile and should be communicated as such.
Practice · Robustness Stress-Test
For each scenario, an analyst has produced an initial result. The widget shows what the result looks like under five different alternative specifications. Your job: judge whether the conclusion is robust (holds broadly), fragile (breaks under several alternatives), or model-dependent (holds only under specific choices). Per-row feedback explains what each alternative reveals.
Honesty About Fragility
Selective reporting is not a robustness check — it is a confirmation exercise dressed up as one.
At this stage, robustness and sensitivity tools are legitimate and necessary. But there is one disciplinary failure that turns these tools against rigor rather than for it: selective reporting.
An analyst runs ten different specifications of a model. Eight produce findings consistent with the analyst's preferred conclusion; two produce findings that contradict it. The analyst presents the eight supportive specifications as a robustness check, omitting the two that disagreed. The analysis looks rigorous — ten specifications, eight in agreement — but the work is dishonest.
The two contradicting specifications were the most analytically informative results, because they revealed the boundary of robustness. Hiding them produces a false sense of strength and exposes the organisation to risk when the conclusion later fails to hold. Selective reporting is not a robustness check; it is a confirmation exercise dressed up as one.
Selective reporting undermines trust in the analytical function. Stakeholders who later discover that contradicting evidence was available but suppressed lose confidence in everything the analyst presents. The disciplined alternative is to report the full sensitivity — "the result holds under specifications A through H but not under I and J, which means the conclusion is conditional on the assumption that…" — preserving credibility through transparency rather than risking it through omission.
Communicating uncertainty without losing credibility
Many analysts believe that admitting uncertainty weakens their position with stakeholders. The opposite is true. Stakeholders develop sharper instincts for fragile claims over time. The analyst who presents a single confident result without examining alternatives is eventually caught out when reality fails to cooperate. The analyst who presents a result with calibrated confidence — including the conditions under which the result would not hold — is trusted precisely because the limits are visible.
Lead with the strength of the result
State the main finding clearly. Don't bury it under caveats so dense that stakeholders cannot extract a recommendation. "The data supports the hypothesis that X increases Y by approximately Z, after testing the main alternative specifications."
Then disclose the boundaries
Name the conditions under which the result would weaken or vanish. "This holds across all customer tiers and time windows tested, but weakens by approximately half if the seasonal control is removed — meaning the magnitude is conditional on accounting for Q4 promotions."
Recommend with calibrated confidence
Tie the recommendation to the strength of the evidence. "Given the robustness of the finding, we recommend X — with a re-check scheduled at Y if condition Z changes." Stakeholders see exactly when to re-examine the decision.
Closing the legitimacy ladder
Module 04 closes the tool legitimacy progression that has built across the programme. Cards A–B forbade tools. Card C permitted only descriptive ones. Cards D–F opened the inferential family conditional on disciplined structure and hypotheses. Cards G–I narrowed the set of hypotheses worth modelling. Cards J–K disciplined the modelling itself. Card L closes the loop: every result the analyst produces is tested against its own fragility before it leaves the desk. This is what rigor looks like in practice.