What separates work that survives challenge from work that only looks like it would.
Three cards on what separates work that survives challenge from work that only looks like it would.
15 minutes of arithmetic that can prevent weeks of confidently wrong analysis.
An analyst at a telecom operator is asked to investigate a 6% decline in quarterly service revenue. A stakeholder proposes: "It's the new prepaid bundle redesign that launched in week three — subscribers are confused and not finding the right plan." Before building any model, the disciplined analyst runs a quick check:
15 minutes of arithmetic that can prevent weeks of confidently wrong analysis.
An analyst at a telecom operator is asked to investigate a 6% decline in quarterly service revenue. A stakeholder proposes: "It's the new prepaid bundle redesign that launched in week three — subscribers are confused and not finding the right plan." Before building any model, the disciplined analyst runs a quick check:
The check did not prove the redesign had zero effect; it almost certainly had some. What it established is that the redesign cannot be the dominant cause of the 6% decline.
It does not produce a precise answer; it produces a precise rejection — in 15 minutes, before any model.
Pick the value the math actually allows at each step.
Willingness to use the simplest tool that answer questions is analytical maturity.
| Tool | What it does | When to reach for it |
|---|---|---|
| Back-of-the-envelope calculations | Quick arithmetic that establishes whether a proposed effect could plausibly produce the observed outcome. Order-of-magnitude reasoning, not precision. | Always, before any formal model. The default first move on any new hypothesis. |
| Rough scenario tables | Simple matrices examining outcomes under best-, worst-, and base-case assumptions for the main parameters. | Testing whether a hypothesis is consistent with observed magnitudes under different assumptions. |
| Simple ratios & rates | Per-subscriber, per-period, per-unit comparisons that ground the analysis in interpretable benchmarks. | Sanity-checking whether the implied behaviour is reasonable at the unit level. |
| Spreadsheet logic checks | Light models that test whether the proposed mechanism produces the observed effect when reasonable inputs are applied. | When a single calculation is too coarse but a full model is still premature. |
They formalise assumptions then estimate parameters in the framework they create.
In practice, models do not reveal reality.
They encode assumptions about how reality works — then estimate parameters within the framework those assumptions create.
Linear regression assumes effects add up in straight lines.
Tree methods assume the data carves into rectangular regions.
None of these are facts about the world; they are choices about how to represent it.
More sophisticated models do not eliminate assumptions — they merely make them harder to see.
A random forest assumes the data can be partitioned by axis-aligned splits. A gradient-boosted model assumes errors can be successively corrected by additive weak learners. A neural network assumes the function lies within the class its architecture can represent.
These assumptions are less visible than those of linear regression — which makes them more dangerous, not less. The disciplined analyst asks the same question regardless of sophistication: what is this method assuming, and would those assumptions hold for my problem?
An analyst models how subscriber demand responds to tariff changes using linear regression. The model estimates a $1 increase reduces demand by 4 units. The coefficient is significant, the R² acceptable. The output looks clean.
In reality demand is highly nonlinear: at low prices subscribers are insensitive; at moderate prices sensitivity grows; at high prices demand collapses as they switch to competitors. The linear model averaged across these regimes — "we can raise prices by $3 with limited impact" may be dangerously incorrect. The math was right; the choice of model was wrong.
Match the model to the question — not the question to the model.
| If the question is about… | Outcome type | Appropriate model family | Key assumptions |
|---|---|---|---|
| How much (continuous magnitude) | Continuous variable (revenue, cost, time) | Linear regression, generalised linear models | Linearity of relationships, additivity, stability across the predictor range |
| Whether (binary classification) | Yes/no, will/won't, has/hasn't | Logistic regression, classification trees | Probabilities are the appropriate target; predictors influence log-odds linearly |
| When (timing of event) | Time-to-event with possible censoring | Survival analysis (Cox models, Kaplan-Meier) | Hazard rates can be modelled; censoring is non-informative |
| Which group (multi-class) | Categorical with multiple classes | Multinomial regression, classification methods | Categories are mutually exclusive; a probabilistic structure is appropriate |
When the analyst finds themselves saying "let's just dichotomise the outcome" or "let's just assume linearity for now" — that is a signal to step back. The model is selected because it matches the question, not because it is convenient.
For each business question, pick the model family, the most consequential assumption it embeds, and what would invalidate that assumption.
A coefficient is not a finding by itself — it is a finding under a model's assumptions.
A coefficient of 0.34 with p<0.001 is not a finding by itself. It is a finding under the model's assumptions. If they hold, the coefficient is meaningful. If they do not, it is an artefact of the model rather than a fact about the world.
| Unconditional language | Conditional language |
|---|---|
| "The model proves that pricing drives churn at 4.2% per dollar increase." | "Under the assumptions of the linear model, the data is consistent with a churn effect of 4.2% per dollar, within the observed price range. If the relationship saturates above $X, the estimate would shift downward." |
| "The data shows that account-care calls drive renewal." | "Holding the observable controls fixed, accounts receiving more calls renewed at a higher rate. Whether this is causal depends on whether call selection is independent of unobserved retention factors." |
| "The algorithm found three segments." | "Under the clustering algorithm's distance metric and chosen cluster count, the data partitions into three groups. A different metric or count would produce a different partition." |
When stakeholders later discover that conclusions depended on assumptions the analyst did not realise they were making, trust in the analytical function erodes. Communicating findings as conditional is harder to deliver and easier to defend.
Strong analysis treats initial results not as findings, but as claims that must survive challenge.
This card addresses a common failure: stopping analysis as soon as a model produces a "good" result. Once a model appears significant or performs well on a chosen metric, the work is considered complete. This feels efficient and produces results that look rigorous — but it creates a dangerous illusion of certainty.
The trap is structural. Modelling tools produce outputs whether or not the underlying analysis is sound. A regression with significant coefficients looks identical to one whose coefficients would vanish under a slightly different specification. The visible output is the same; the underlying robustness is not.
Three concepts that look similar — conflating them is why fragile findings get presented with unwarranted confidence.
| Concept | What it asks | What it can't tell you |
|---|---|---|
| Confirmation | Does the data support this hypothesis under one specific setup — one model spec, one time window, one sample, one set of assumptions? | Whether the support would persist if the setup changed. It establishes a result is possible, not robust. |
| Robustness | Does the support persist when reasonable alternatives are considered? Do different specifications, samples, or windows produce broadly similar conclusions? | Whether the result is causally correct. It is about stability across analytical choices, not ruling out confounders. |
| Sensitivity analysis | How much do conclusions change when key assumptions, parameters, or inputs vary? Which inputs are the conclusions most dependent on? | Whether the variations tested cover the full space of plausible alternatives. It only tests what the analyst chooses to vary. |
The willingness to attack one's own results feels like betraying the work already done. Card L reframes it: a result that has survived challenge is far more defensible than one that has not. The analyst who openly stress-tests their findings demonstrates more rigor, not less. The three concepts together form a layered defence against fragility — skipping any layer leaves the analysis exposed.
When one specification carries the result, the conclusion is about the modelling decision — not about the world.
When conclusions rely heavily on one specific choice — a particular regression specification, time window, variable definition, or handling of missing data — the analysis becomes vulnerable in a specific way. The result holds, but only because of the choice. A reasonable alternative would produce a different result.
An analyst builds a regression finding a strong relationship between marketing spend and sales. The coefficient is precise, the p-value small. Before recommending a 30% spend increase, the analyst tries an alternative: defining spend by month rather than week, and adding a quarterly seasonality control. The relationship vanishes. The original finding was an artefact of the time aggregation and missing control.
Model dependence is often invisible to the analyst who created the model — the result feels stable because it has not been challenged. Challenge is exactly what reveals it. The remedy is not more sophisticated models; it is multiple specifications, checked for agreement.
The goal is not to optimise results — it is to find the specifications under which the answer changes.
| Tool | What it does |
|---|---|
| Alternative model specifications | Re-run the analysis with different functional forms, control sets, or model families. If the conclusion is robust, it survives reasonable alternatives. |
| Variable transformations | Test whether log-transforming variables, using percentages instead of levels, or different aggregations changes the conclusion. Many findings appear or disappear depending on these. |
| Different time windows or samples | Examine whether the result holds when restricted to a subperiod, a different subset of subscribers, or a different region. A finding that holds only in one slice is by definition fragile. |
| Sensitivity tables & scenario checks | Examine how outputs change as key parameters vary across plausible ranges. The output is not a single number but a sensitivity surface. |
| Regression diagnostics & assumption toggles | Formal checks of model assumptions, with deliberate exploration of what changes when assumptions are relaxed or modified. |
The analyst is not looking for the specification that produces the most favourable answer — they are looking for the specifications under which the answer changes. A result that survives many such tests is genuinely strong. A result that breaks under several is fragile and should be communicated as such.
An initial result is shown under five alternative specifications. Judge whether the conclusion is robust, fragile, or model-dependent.
Selective reporting is not a robustness check — it is a confirmation exercise dressed up as one.
An analyst runs ten specifications. Eight produce findings consistent with the preferred conclusion; two contradict it. The analyst presents the eight as a robustness check, omitting the two that disagreed. The analysis looks rigorous — ten specifications, eight in agreement — but the work is dishonest.
The two contradicting specifications were the most analytically informative results — they revealed the boundary of robustness. Hiding them produces a false sense of strength and exposes the organisation to risk when the conclusion later fails to hold.
Many analysts believe admitting uncertainty weakens their position. The opposite is true. The disciplined alternative: lead with the strength of the result, then disclose the boundaries — name the conditions under which it would weaken or vanish — and recommend with calibrated confidence, tying the recommendation to the strength of the evidence.
Module 04 closes the tool legitimacy progression that has built across the whole programme.
Cards A–B forbade tools. Card C permitted only descriptive ones. Cards D–F opened the inferential family conditional on disciplined structure. Cards G–I narrowed the set worth modelling. Cards J–K disciplined the modelling itself. Card L closes the loop — this is what rigor looks like in practice.
Three questions across Cards J and K. Select the best answer — feedback appears immediately.
Two questions on Card L — completing the module assessment.
You've worked through the full Rigor module — Card J's validation discipline, Card K's assumption-aware modelling, and Card L's robustness orientation. The full legitimacy ladder is now complete: judgment → structure → focus → rigor. Module 05 picks up the surviving conclusions and teaches the discipline of going beyond correlation to causation — the depth work that produces decisions you can defend.