How to Read a Study Without Believing It or Dismissing It

This is the lesson the whole Part has been for.

You now have the philosophy, the design principles, the sampling logic, the two toolkits, the analytic machinery, the fallacies, the ethics and the crisis. The remaining task is to turn all of it into something you can actually do — with a paper open, a limited amount of time, and no intention of becoming a statistician.

The goal is not a verdict. It is a proportionate belief: this is one moderately well-designed study suggesting a small effect in a specific population, consistent with two others and inconsistent with one. That sentence is worth more than either "science says" or "you can prove anything with statistics", and it is what the rest of this lesson teaches you to write.

A headline, a press release, a paper, and three different findings.

The headline: "Free school breakfasts boost children's grades, study finds."

The press release, from the university: a large study of a breakfast programme found that pupils in participating schools achieved better results, with the effect described as "substantial", and a quotation from the lead author about the importance of nutrition for learning.

The paper takes eleven minutes to change the picture, and it does so twice.

First read: the design is good. The programme was rolled out to schools in a staggered order, and the researchers exploited that: schools that had not yet received it act as the comparison, and the analysis compares changes rather than levels (see 7.4.3). They show four years of pre-programme trends for both groups, moving together. The identifying assumption is not merely asserted. They pre-registered the primary outcome. They report intention-to-treat.

So this is not a study to dismiss. It is better than most.

Second read: the finding is smaller and narrower than the headline.

The effect on the pre-registered primary outcome — the main attainment measure across all pupils — is positive, small, and its confidence interval includes values close to zero. In natural units, roughly the equivalent of a few weeks of additional progress.

The large effect in the press release is a subgroup: pupils eligible for free school meals, where the estimate is around three times larger. The paper is careful about this — it reports the interaction test, notes that the subgroup analysis was pre-specified, and says the estimate is imprecise. The press release is not careful about it, and the headline is a subgroup finding presented as a general one.

And a third thing, which the paper reports in a single sentence in the results and which nobody picked up. Attendance rose more than attainment did. The mechanism may be that children come to school, rather than that they eat. If so, a programme delivering food later in the day would not work, and a free non-food attraction might.

Now the proportionate summary, which is the output of reading properly:

A well-designed staggered-rollout evaluation finds a small positive average effect on attainment, imprecisely estimated, with a larger pre-specified effect among poorer pupils. Attendance moved more than attainment, suggesting the mechanism may be attendance rather than nutrition. The headline generalises a subgroup finding.

Three sentences. No statistics computed. Eleven minutes. And it is more useful than either believing the headline or dismissing it.

The procedure

Step 0 — find the actual study, and note who is talking.

Press releases exaggerate, and the exaggeration usually starts there rather than in the newsroom. A systematic study of hundreds of UK university press releases and the news coverage they generated found that a large share contained claims stronger than the underlying paper supported — advice not given in the paper, causal language for correlational findings, and inference to humans from animal studies — and that exaggerated news coverage was overwhelmingly preceded by an exaggerated press release. The same work found that exaggerating did not even gain more coverage.

So: find the paper. If you cannot, that is itself information about how much weight to place on the claim.

Step 1 — what kind of question is this?

Descriptive, relational, causal, mechanistic, interpretive or evaluative (see 7.2.1). This determines what evidence is required, and the commonest defect in the world is relational evidence carrying a causal conclusion.

Step 2 — what is a case?

A person, a household, a school, an area, a year, a person-year, an event (see 7.6.4). This takes five seconds and prevents the ecological and individualistic fallacies outright.

Step 3 — who is in the data, and who could not be?

The frame, the selection, the response rate, the attrition, the filtering by outcome (see 7.3.1, 7.3.2). Not "how many" — who, and who is missing, and would the missing answer differently?

Step 4 — how was the key thing measured?

The exact question, the exact indicator, the exact threshold (see 7.2.2). A surprising number of astonishing findings are ordinary findings about an unusual measure.

Step 5 — compared to what?

Find the comparison group, or note that there isn't one (see 7.4.2). If the comparison is people who did not take part, ask what made them different.

Step 6 — what assigned the treatment?

Randomisation, a lottery, a threshold, a policy date, or a choice (see 7.4.3). This single fact predicts most of what can be concluded, and if the answer is "the participants chose", the study is descriptive whatever its verbs say.

Step 7 — how big, and how uncertain?

In natural units, with an interval (see 7.6.1, 7.6.3). Ask whether the smallest value in the interval would still interest you.

Step 8 — could it have come out otherwise?

Was the outcome pre-specified? How many outcomes, subgroups and time points were examined? Is the headline finding the primary one (see 7.2.3, 7.6.3, 7.8.2)?

Step 9 — whose account is this, and what is invisible?

Who produced it, for whom, funded by whom, and what does this design structurally prevent from being seen (see 7.1.2, 7.1.3)?

Step 10 — is this one study or a body of work?

Almost never update strongly on one study. Has it been replicated? Does it agree with well-identified work using different designs?

In twenty minutes you will get through most of these. In five, do steps 5, 6 and 7 — the comparison, the assignment and the size — because they carry the most weight per second spent.

How to read the paper itself

Not front to back. The sections are not equally informative and one of them is the authors' hopes.

Title and abstract — the authors' preferred framing. Note the verbs. "Associated with", "linked to", "predicts" in the abstract and "causes", "leads to", "drives" in the discussion is the single most common tell in the literature, and it is worth checking every time.

Introduction — the case for the question, with a literature reviewed selectively. Useful for context, not for judging the study.

Methods — the actual study. Read this properly. Everything in the procedure above is answered here or nowhere. If it is vague, that is the finding.

Results — read the tables and figures, not the prose. The text tells you which results the authors want you to notice; the tables tell you what there is. Look for outcomes reported in a table and not mentioned in the text.

Limitations — read this early, and then ask what is not in it. Every limitations section lists the concerns the authors were comfortable naming. The one they did not name is usually the one that matters — and a limitations section that identifies the genuine threat to the study's central claim is a strong signal of a serious paper.

Discussion — read last, sceptically. This is where the modest finding acquires implications for policy.

And check funding, conflicts, and pre-registration. Funding is a reason for heightened scrutiny of the design, not a refutation (see 7.1.2). A pre-registration link is worth more than any single design feature, because you can compare what was planned with what was reported.

For qualitative work, a different set — because the criteria are different, not absent.

Who was studied, and by what logic? Theoretical, purposive, comparative — or convenience with no account (see 7.3.1).

How much contact, over how long, in what role, admitted by whom? (See 7.5.1.)

What was the analytic procedure? Named and described specifically enough that you can picture it happening (see 7.7.1).

Are the themes claims or topics? If the headings are noun phrases, the analysis has not happened.

Is there anything that complicates the argument — a negative case, a participant who did not fit, a moment when the analysis changed? This is the strongest single signal in qualitative work.

And is it illuminating? This is not proceduralisable and it is not optional (see 7.7.2).

Do not ask a qualitative study for a sample size justification in statistical terms, an effect size, or a control group. Those are category errors, and making them marks you as someone who cannot assess the work rather than as someone being rigorous.

Tells, in both directions.

Weak work announces itself.

No comparison group. A long list of controls with no account of why those. Outcomes not pre-specified, and a headline result buried at position eleven. Percentages with no denominators. A subgroup finding in the title. Confidence intervals absent. Sample sizes only in a supplement. Causal verbs in the discussion that the design cannot support. A press release stronger than the paper. And — the tell that costs nothing to check — "further research is needed" as the only limitation.

Strong work also announces itself, and this list is worth as much as the other.

A pre-registration link, with the reported outcomes matching. Null results reported alongside positive ones. A limitations section that names the real threat rather than a decorative one. Sensitivity analyses showing how much confounding it would take to overturn the result (see 7.6.2). Effect sizes in units a reader can judge. Data and code available. Pre-trends plotted rather than asserted. And, most tellingly of all: a sentence somewhere stating what would have counted as evidence against the authors' claim.

That last one is rare, and when you find it, you are almost always reading someone careful.

For a news story, when you are not going to read the paper.

One — is there a study, and can you find it? A named journal and authors, or a "report" from an organisation with an interest.

Two — humans, and which ones? Animals, students, volunteers, one country, one age group — reported as though about everyone.

Three — how many, and who was missing?

Four — compared to what? If no comparison appears anywhere, the causal claim has no basis.

Five — absolute numbers. "Doubles the risk" of what, from what? (See 7.6.1.)

Six — who paid, and does the press release say more than the study?

Six questions, no expertise, most of the value. And the most useful habit of all is the cheapest: when a finding delights you, slow down. The results that confirm what you already believe are the ones you will check least and should check most — which is 7.1.2's strong objectivity, applied to yourself, in the ten seconds before you share something.

Because the alternative to this skill is not neutrality. It is being governed by whoever presents evidence most confidently.

Every argument you will encounter about schools, crime, migration, health, work, housing, technology and inequality is conducted with evidence. Some of it is good. A lot of it is a subgroup finding from an underpowered study with a good communications department. And the difference is visible in twenty minutes to anyone who knows what to look at.

Three things this Part has been arguing, brought together.

Every method answers a specific question and is blind to others. There is no method that establishes prevalence and mechanism and meaning and cause. The first move in reading anything is to identify what this design could and could not have found.

Every finding was produced by people, in institutions, with incentives (see 7.1.2, 7.8.2). Not a reason for cynicism — a reason to look at the procedures that make claims checkable, which is what objectivity realistically consists of.

And the point of all of it is to be able to hold a belief in proportion to its evidence — which is harder than either believing or refusing, and is the only thing that lets you change your mind for good reasons rather than social ones.

One last connection, back to where the course began. Part 1 called the sociological imagination a habit of seeing personal troubles as public issues, and named debunking as one of its core moves (see 1.1.1, 1.5.2). Debunking without method is just a different set of beliefs, held with the same confidence and no more warrant.

Part 7 is what makes the debunking honest. Everything after this Part — the substantive sociology of inequality, institutions, deviance, change and public life — rests on claims made with these tools. You can now check them, including this course's own.

The goal is a proportionate belief, not a verdict: one moderately designed study, a small effect, a specific population, consistent with some work and not other work.

The worked example: a well-designed staggered-rollout evaluation with plotted pre-trends and a pre-registered primary outcome, whose average effect was small and imprecise, whose headline finding was a subgroup, and whose most interesting result — attendance moving more than attainment, implying a different mechanism — appeared in one sentence and was reported by nobody.

The procedure, in order of yield. Find the paper, not the press release — exaggeration in health news is overwhelmingly preceded by exaggeration in the university's own release. Then: what kind of question; what is a case; who is in the data and who could not be; how the key thing was measured; compared to what; what assigned the treatment; how big and how uncertain; could it have come out otherwise; whose account is this and what is invisible; and is this one study or a body of work. With five minutes, do comparison, assignment and size.

Read the paper out of order. The abstract is the authors' framing — check whether the verbs change between abstract and discussion. The methods are the study. Read the tables, not the results prose. Read limitations early and ask what is missing from them. Read the discussion last and sceptically.

Qualitative work takes a different six questions — sampling logic, contact and role, analytic procedure, themes as claims, anything that complicates the argument, and whether it illuminates. Demanding a control group of an ethnography is a category error, not rigour.

Weak work announces itself: no comparison, unexplained control lists, unspecified outcomes, subgroup headlines, missing intervals, causal verbs the design cannot carry, a press release stronger than the paper. Strong work announces itself too: pre-registration matching the report, null results included, a limitations section naming the real threat, sensitivity analysis, natural units, open data, plotted pre-trends — and, rarest and best, a statement of what would have counted as evidence against the claim.

And when a finding delights you, slow down.

Proportionate belief — a claim held with the strength its evidence supports, stated with its conditions.

Press-release exaggeration — the documented origin of most overstatement in science news.

Primary outcome — the pre-specified main result, against which the headline should be checked.

Subgroup finding — an estimate for part of the sample, imprecise and routinely generalised.

Verb drift — associational language in the abstract becoming causal language in the discussion.

Pre-registration check — comparing what was planned with what was reported.

Sensitivity analysis — how strong unmeasured confounding would have to be to overturn a result.

Category error in appraisal — applying one tradition's criteria to another's work.

Generic scepticism — a stock objection applied without checking whether it bears on this study.

One — run the full procedure once. Take any study behind a news story this week and work through the ten steps. Write the three-sentence proportionate summary at the end. The first time will take an hour; the fifth will take fifteen minutes.

Two — check verb drift. For five papers, compare the verbs in the abstract with the verbs in the discussion. Note how often the claim strengthens without new evidence.

Three — read a limitations section for what is absent. Take a study and write down the biggest threat to its central claim. Then check whether the authors listed it.

Four — compare a press release with its paper. Find both for the same study. Mark every claim in the release that the paper does not support.

Five — do it to something you agree with. Take a finding you have cited or shared because it confirmed something you believe, and run the procedure on it properly. This exercise is the whole of Part 7 in one act, and it is the only one on this list that is genuinely uncomfortable.