Significance, Effect Size and What p Actually Means

A finding is announced as statistically significant, and everyone relaxes. The number has done its job: it has certified that this is real.

It has not. The p-value is a genuinely useful quantity that almost nobody — including a large proportion of professional researchers — can define correctly, and the gap between what it means and what people take it to mean is responsible for a substantial share of the false things the public believes about the social world.

This lesson is the most technical in Part 7 and it requires no mathematics. Everything here can be understood in words, and once understood it cannot be unseen.

Two demonstrations, one embarrassing and one funny.

The first. Researchers gave a short questionnaire to three groups: psychology students, university lecturers who taught psychology but not statistics, and lecturers who taught the statistics and methods courses themselves.

The questionnaire described a simple result — a significant difference between two groups, p < 0.01 — and offered six statements about what this meant. All six statements were false. Participants were asked to mark each as true or false.

Every single student endorsed at least one false statement. Around 90 per cent of the psychology lecturers did. And roughly 80 per cent of the people who taught statistics did too.

Sit with that last figure. The most commonly endorsed errors were the obvious, tempting ones: that the p-value gives the probability that the null hypothesis is true, that it gives the probability the result was due to chance, and that it tells you how likely the finding is to replicate. None of these is what a p-value is, and the people teaching it thought they were.

The second demonstration is a joke made rigorously.

Three researchers set out to show how easily a false finding can be produced using only ordinary, undeclared, entirely common analytic choices. They ran real experiments on real participants.

Their published result: listening to the Beatles' "When I'm Sixty-Four" made participants a year and a half younger. Not feel younger. Their reported date of birth indicated they were chronologically younger — which is impossible, and was the point.

How they did it, with nothing that would look wrong in a methods section: they collected several outcome measures and reported the one that worked; they collected several covariates and included the one that helped; they tested after every ten participants and stopped when it reached significance; and they dropped one condition from the write-up. Each choice is defensible in isolation. Together they made a p-value below 0.05 nearly certain.

They called this "researcher degrees of freedom", and their demonstration is the reason pre-registration exists.

So what is it, then?

A p-value is the probability of getting data at least as extreme as what you observed, IF the null hypothesis were true — and if every other assumption in your model were also true.

Read the conditional clause carefully, because everything follows from it. The p-value assumes the null is true and then asks how surprising your data would be. It is a statement about data given a hypothesis.

People want the reverse: the probability of the hypothesis given the data. That is a different quantity, it cannot be obtained by turning the first one round, and getting it requires something the p-value does not contain — how likely the hypothesis was to begin with.

Four things a p-value is not, and each is endorsed by most people who use them.

Not the probability that the null hypothesis is true. p = 0.03 does not mean a 3 per cent chance there is no effect.

Not the probability the result is "due to chance". Same error in friendlier clothing.

Not the probability the finding will replicate. A study with p = 0.05 has, roughly, a coin-flip chance of reaching significance again in an identical replication.

And not a measure of size or importance. A p-value is a function of the effect size and the sample size and the noise. With 200,000 observations, a difference too small to matter to anyone will be highly significant. With 40, a large and important difference will not be.

The base rate: why significant findings are wrong more often than 5 per cent

The single most important idea in this lesson, worked through with numbers.

The 5 per cent threshold means: if the null is true, you will wrongly declare significance 5 per cent of the time. That is the false-positive rate among true nulls.

It is not the rate at which significant published findings are false, and the difference is enormous. To get that, you need two more things: how often the hypotheses being tested are true, and how good the studies are at detecting real effects when they exist — the power.

Work through a field where 10 per cent of tested hypotheses are true, studies have 50 per cent power, and the threshold is 5 per cent. Imagine a thousand hypotheses tested.

100 are true. With 50 per cent power, 50 of them produce a significant result. The other 50 real effects are missed.

900 are false. At a 5 per cent false-positive rate, 45 of them produce a significant result anyway.

So 95 significant findings are produced, and 45 of them — nearly half — are false.

Nothing in this calculation involves any misconduct. Every study was run honestly at the standard threshold. The false-discovery rate is high because the base rate of true hypotheses is low and the power is mediocre, which describes a great deal of social science.

And it gets worse in exactly the ways you would expect. Lower power makes it worse. Testing more speculative hypotheses makes it worse. Any flexibility in analysis raises the effective false-positive rate above 5 per cent. Publication bias then selects the significant results for publication (see 7.1.2), so the published literature is enriched in exactly the wrong direction.

This is the argument behind the claim, made forcefully in 2005 and widely accepted since, that a large share of published research findings in many fields are false. It is not cynicism. It is arithmetic.

Power, and the trap in an underpowered success

Power is the probability of detecting an effect that is really there.

It depends on the size of the real effect, the sample size, the noise in your measures, the design, and the threshold. Small effects, small samples and noisy measures — the standing condition of social research — give low power.

Surveys of published work have repeatedly found typical power well below 50 per cent for the effect sizes actually being studied. Which means the median study is more likely to miss a real effect than to find it.

The obvious consequence is missed findings. The non-obvious consequence is far more damaging, and it is the thing to remember from this section.

In an underpowered study, the significant results are exaggerated.

Why: to clear the significance threshold with a small sample, the observed effect has to be large. Real effects plus noise sometimes produce large observed values and sometimes small ones. Only the large ones get through the filter. So the estimates that survive are, on average, bigger than the truth — sometimes several times bigger. This is the magnitude error — often called the winner's curse.

And in the worst cases the surviving estimate points the wrong way. When power is very low, a meaningful fraction of the significant results have the wrong sign — reporting a benefit where there is a harm, or vice versa.

The practical rule is counterintuitive and correct: a surprising, large, significant effect from a small study is not strong evidence. It is the expected output of a weak design. The appropriate response is not excitement but a demand for replication with adequate power (see 7.8.2).

Effect size, and the question significance never answers

"Is there an effect?" is nearly always the wrong question. "How big, and how uncertain?" is the right one.

Report estimates in natural units wherever possible. Three fewer days of absence. Four percentage points more likely to enrol. £1,200 a year. These can be judged by a person who knows the field, which standardised measures cannot.

Standardised effect sizes — Cohen's d, correlations, odds ratios — are needed to compare across studies with different measures, and they carry a hazard: the conventional labels for "small", "medium" and "large" were offered as rough placeholders by their author, explicitly as a last resort, and have hardened into a rubric. A "small" effect on a population-wide exposure can matter more than a "large" one in a niche.

And report the confidence interval, not just the point estimate. The interval carries the information the p-value discards: the range of effects compatible with your data. A study reporting "significant, p = 0.04" with an interval running from "trivially small" to "implausibly enormous" has told you very little, and the p-value concealed that while the interval reveals it.

The habit worth building is to read the interval and ignore whether it crosses zero. Ask instead: is the whole interval large enough to matter? Is it all too small to matter? Or does it span both, in which case the study is uninformative for practical purposes — regardless of what the p-value said.

Multiple comparisons, and the version you cannot see.

The visible problem. Test twenty independent hypotheses at the 5 per cent threshold with nothing real going on, and you expect one significant result. Test a hundred, expect five. Any study reporting many outcomes, many subgroups or many time points has this problem, and the standard corrections address it: the very conservative Bonferroni adjustment, or the more sensible false discovery rate approach, which controls the expected proportion of false positives among the results declared significant.

The invisible problem is worse and has no correction.

You do not have to run many tests to have many potential tests. A researcher analysing a dataset makes dozens of reasonable decisions: which outcome, which subgroup, whether to exclude outliers, how to code a variable, whether to log a skewed measure, which covariates, which time window. Each is made after seeing the data, and each could have gone otherwise.

Even if only one analysis is ever run, the result was chosen from a garden of forking paths — and its true false-positive rate is that of the whole garden, not of the single path walked. No statistical correction can fix this, because the alternative analyses were never performed and cannot be counted.

This is why pre-registration is the answer and post-hoc correction is not. Specifying the analysis before seeing the data collapses the garden to a single path (see 7.4.2, 7.8.2).

And p-hacking leaves fingerprints. Across a literature, the distribution of reported p-values should decline smoothly. An unusual bump just below 0.05 — many results at 0.045 and few at 0.055 — is the signature of choices made to get across the line, and this pattern has been documented in several fields.

Four verbal errors that occur constantly, including in journals.

"Approaching significance", "trending towards significance", "marginally significant". p = 0.07 is not almost significant; the threshold is arbitrary, and this phrasing is a way of claiming a result while retaining deniability. The honest report is the estimate and its interval.

"No significant difference" reported as "no difference". Absence of evidence again (see 7.2.3). The correct statement names what the study could have detected: this study could not detect an effect smaller than X. If you actually want to claim no meaningful effect, there is a procedure for it — equivalence testing, which tests whether the effect is smaller than a threshold you specify in advance — and it is almost never used because it requires committing to what "meaningful" means.

Comparing significance instead of testing the difference. The intervention worked in men (p = 0.01) and not in women (p = 0.12), therefore it works differently by sex. This does not follow. The difference between a significant result and a non-significant one is not itself significant, and the comparison must be tested directly, as an interaction. This error is extremely common in subgroup reporting and it is how spurious "it only works for group X" claims are manufactured.

Dichotomising everything. Reducing a rich estimate with an interval to "significant / not significant" throws away almost all the information — and produces the absurdity of two studies with nearly identical estimates being described as contradicting each other because one interval just crossed zero and the other just did not.

What the field is doing about it, stated evenhandedly.

In 2016 the American Statistical Association issued a formal statement on p-values — an unusual step — setting out that p-values do not measure the probability that a hypothesis is true or the importance of a result, that scientific conclusions should not be based only on whether a threshold is passed, and that proper inference requires full reporting and transparency.

In 2019 a large group of scientists went further, calling in Nature for the abandonment of "statistical significance" as a category — not of p-values, but of the dichotomy — and were joined by hundreds of signatories. Their argument: thresholds convert continuous evidence into a false binary, which produces both overclaiming and the misreading of null results as evidence of absence.

And the counterargument deserves stating, because this course applies the symmetry rule. Decisions are binary. A drug is licensed or not; a policy is adopted or not; a paper is published or not. A framework with no threshold anywhere leaves the decision to be made by unstructured judgement, which is not obviously an improvement and may simply relocate the arbitrariness somewhere less visible. Critics of abolition also note that without any calibrated rule, claims will be made on the basis of whatever the author found convincing.

Where the reasonable middle sits: report estimates and intervals as the primary result; use p-values as one continuous piece of evidence rather than a verdict; pre-register the confirmatory analysis and label everything else exploratory; and where a threshold is genuinely needed for a decision, set it deliberately for that decision rather than inheriting 0.05 from a convention Fisher offered as a convenience.

Because "statistically significant" is doing work in public argument that it cannot support.

Five questions for any quantitative finding.

How big is it, in units I understand? If the report does not say, that is the first thing to notice.

What is the interval? Does the whole range matter, none of it, or does it span both?

How many things were measured? Outcomes, subgroups, time points, model variants.

Was the analysis specified in advance? If not, treat the result as a hypothesis rather than a finding.

And was this study capable of detecting an effect of the size claimed? If it was small and the effect is startling, expect the estimate to be an exaggeration.

And a final note about what all this is for. The purpose of understanding p-values is not to acquire a stick for beating quantitative work. It is to know what the evidence supports, which is nearly always something more modest and more useful than the headline. The finding in the story — that a song made people younger — was produced honestly, by real researchers, using techniques in daily use.

They published it to show you what a significant result can be worth.

A p-value is the probability of data at least as extreme as observed, if the null hypothesis and all the model's assumptions were true. It is a statement about data given a hypothesis, and cannot be turned round.

It is not the probability the null is true, not the probability the result is "due to chance", not the probability of replication, and not a measure of size or importance. In one study, roughly 80 per cent of the lecturers who taught statistics endorsed at least one of these errors.

The false-positive rate among true nulls is not the rate at which significant findings are false. With 10 per cent of hypotheses true and 50 per cent power, of 95 significant results about 45 are false — arithmetic, not cynicism — and publication bias then selects for exactly those.

Power is the chance of detecting a real effect, and it is typically low. The non-obvious consequence: significant results from underpowered studies are exaggerated, sometimes several-fold, and at very low power a meaningful share have the wrong sign. A surprising large significant result from a small study is the expected output of a weak design.

Report effect sizes in natural units and confidence intervals rather than verdicts. Read the interval and ask whether all of it matters, none of it, or whether it spans both — the last meaning the study is uninformative whatever the p-value said.

Multiple comparisons inflate false positives, and the invisible version has no correction: the garden of forking paths means the analysis was chosen from many possible analyses, even when only one was run. Pre-registration is the answer; post-hoc adjustment is not. P-hacking leaves a bump just below 0.05.

Four verbal errors: "approaching significance"; "no significant difference" reported as "no difference" (use equivalence testing if you mean it); comparing significance across subgroups instead of testing the interaction; and dichotomising rich estimates into a binary.

The ASA's 2016 statement and the 2019 call to retire significance are the field's response — and the counterargument that decisions are binary and thresholds must live somewhere deserves equal weight.

p-value — the probability of data at least as extreme as observed, given the null hypothesis and the model's assumptions.

Type I / Type II error — a false positive; a missed real effect.

Power — the probability of detecting an effect that exists. Typically low in social research.

False discovery rate — the proportion of significant findings that are false; depends on base rate and power, not on alpha alone.

Magnitude (Type M) and sign (Type S) errors — exaggerated estimates, and estimates pointing the wrong way, produced by low power plus a significance filter.

Winner's curse — the selection of overestimates by the significance threshold.

Effect size — the magnitude, in natural or standardised units.

Confidence interval — the range of effects compatible with the data; more informative than a verdict.

Researcher degrees of freedom — undeclared analytic choices that raise the effective false-positive rate.

Garden of forking paths — the many analyses that could have been run, making a single reported analysis a selected one.

p-hacking / p-curve — analytic choices made to cross the threshold; the distributional signature they leave.

Bonferroni / false discovery rate correction — conservative and less conservative adjustments for multiple testing.

Equivalence testing — the procedure for actually claiming an effect is negligibly small.

Interaction test — the correct way to compare an effect across subgroups.

One — define it out loud. Say what a p-value is, in one sentence, without notes. Then check it against the definition above. Most people discover they had one of the four wrong versions.

Two — work the base rate. Take a field you are sceptical about. Guess what fraction of its hypotheses are true and what power its studies have, and work out how many of its significant findings would be false. The arithmetic is three lines and it is sobering.

Three — find the interval. Take any reported finding and locate the confidence interval. Ask whether the smallest value in it would still interest you. If not, the finding is weaker than its headline.

Four — count the tests. For any paper with subgroup results, count the outcomes, subgroups and time points reported. Multiply. Then ask how many significant results you would expect from nothing at all.

Five — catch the subgroup error. Find a claim that something "worked for X but not for Y". Check whether the difference between the groups was tested directly, or whether two separate significance verdicts were simply compared. It is usually the second.