Writing the report

Concept

The report is where an analysis becomes a claim made to your colleagues and peers. Up to this point a mistake costs you a wrong number in a notebook. From here onward, a mistake is something you have asserted in public and that someone else may build on.

An AI system will write this document for you, quickly and in the right style and tone. The difficulty is that fluency and accuracy are independent when drafting prose. The sentence "responding was significantly higher under the richer schedule" is well-formed English whether a test was run, a schedule varied, or the difference exists. Everything you validated in lesson 18 was a number, and numbers are checkable. Sentences about numbers are not checkable in the same way, which is why the report needs its own pass.

There are two failure modes. A draft can claim more than the analysis established, and a draft can omit the decisions that make the result interpretable such that a reader can't tell what was actually done. A good report avoids the first and reports those decisions thoroughly.

A note on the data

The figures below come from the simulated cohort in data/synthetic/, analyzed under the decisions made in lessons 14 and 16. In a real manuscript the simulated origin of this data would itself be the first thing disclosed. We keep saying it here for the same reason.

Start from the participant flow

Before any prose, account for every person and every row. A reader must be able to follow the sample from enrollment to analysis without a gap, and you cannot write an accurate methods section from memory.

Ask Claude:

Produce a participant flow accounting for data/synthetic/: how many participants enrolled,
each exclusion applied with its count and reason, and how many participants and response
rows remain for analysis. Report the percentage of raw response rows removed.

What comes back:

Enrolled                                                    40 participants
  - response rows matching no enrolled participant           18 rows removed
  - produced no responses at all                              1 participant
  - automated responder (median gap 97 ms, 2,662 rows)        1 participant
Analyzed                                        38 participants, 9,119 rows
                                        (22.7% of raw response rows removed)

Reading the output: Forty people entered, two are not in the analysis for stated reasons, and 38 remain. Nearly a quarter of the raw rows are gone, almost all of them from the single automated responder. That last figure belongs in the report because it is large. A reader who is told that 22.7 percent of rows were removed can judge the cleaning. A reader who is not told has no way to.

Confirm it by checking that the numbers reconcile in both directions. The rows removed (18 + 2,662 = 2,680) plus the rows analyzed (9,119) must equal the raw total (11,799). If an exclusion is described but its rows are unaccounted for, the flow is incomplete.

The methods section is a record of decisions

Most of what belongs in methods is the set of choices that determine what the results mean. Lessons 13 through 16 generated all of them.

Ask Claude:

Draft a methods paragraph for this analysis. State the analyzed sample, the demographic
summary, the exclusion rules, the definition of the response rate measure including phase
durations, and whether group or per-participant weighting was used. Use only quantities
computed from the data, use the code your created as the guide for what was done and the
sequence it was compelted in, and do not fill any gap with a plausible value.

What comes back:

Analyzed sample: 38 participants (9,119 responses). Demographics were available for
37; one participant's record was missing and one reported age (142) was treated as
missing. Age: M = 34.7, SD = 8.7, range 18-54 (n = 36). Gender after standardizing
spellings: Female 16, Male 13, Non-binary 4, Prefer not to say 4. Education: bachelor's
13, some college 10, graduate 7, high school 6, less than high school 1.
 
Exclusions: responses not attributable to an enrolled participant (18 rows); one
participant with a median inter-response time below 150 ms; one participant who produced
no responses at all. Non-completers (n = 3 of the analyzed sample) were retained,
contributing data for the phases they reached.
 
Measure: responses per minute, computed per participant as the count of clicks on a
given button within a phase divided by that phase's duration (2, 2, and 1 minutes for
phases 1-3). Group values are means across participants, weighting each participant
equally.

Reading the output: notice how much of this paragraph reports a decision. Non-completers were retained. Participants are weighted equally instead of by their number of clicks. The age of 142 was treated as missing instead of the whole record being dropped. Each was a choice, each changes the numbers, and none of them are visible in the results table. A reader can evaluate the analysis only if you tell them what was done.

Where a draft claims more than the analysis supports

The results section is where failure is most costly. The useful exercise is to read a draft adversarially, marking every claim against the analysis that supports it.

A draft written without constraint might contain:

Across 40 participants, responding was significantly higher on the reinforced button
(p < .001), demonstrating that participants preferred the button paying on the richer
FR10 schedule. Response rates declined in phase 3, confirming extinction.

Those three lines contain five problems, and each one is a distinct kind of error.

The sample was 38, not 40. The figure reverted to the enrollment number, which is the most common arithmetic error in a results section.

No test was run, so p < .001 is a fabricated result rather than a reported one. This is among the most damaging things a draft can contain, because a p-value is the one element a reader will not independently recompute.

"Preferred" is a claim. What was measured were clicks. Rate of responding is the finding; preference is an interpretation of it, and the report should keep them separate.

The FR10 schedule did not exist. The deployed task was an FR1 on the currently active circle, as lesson 12 established. This claim is the one an AI system is most likely to produce, because the study description said so and the description is the more prominent document. It is also the only error here that would survive a careful reading by someone who knew the analysis but not the experimental task.

"Confirming extinction" overstates a single observation. The decline is consistent with extinction. Confirmation would require ruling out alternative hypotheses such that responding declined from fatigue because participants were 4 minutes into a task, clicking rapidly throughout. This experimental design can't rule out many alternative hypotheses.

A corrected version:

Across 38 participants, clicks were allocated to whichever circle currently contacted
reinforcement on a FR1 schedule. Mean response rate on the reinforced circle was 43.6 per 
minute in phase 1 and 43.9 in phase 2, compared with 5.3 to 6.6 per minute for the two 
circles that did not contact reinforcement in those conditions, respectively. The same 
pattern was present in individual participants. When no circle contacted points in phase 3,
rates on both target circles fell to 5.7 and 5.2 per minute (n = 34-35), at or below the
distractor's rate of 8.7, a decline consistent with extinction.

Confirm it by requiring a source for each sentence. Take the draft and ask for the specific computation behind every claim in it.

Ask Claude:

For each sentence in this draft, cite the specific number or analysis that supports it.
Mark any sentence that is an interpretation rather than a finding. Mark any sentence you
cannot support from the analysis we ran.

Cut every sentence that survives this is unsupported. Claiming less than the reader hoped for does not weaken a report. Claiming more than the data support does weaken a report because those are the claims that will not replicate.

The limitations are part of the result

A limitations paragraph tells a reader what the study can't answer, which is often what they most need to know.

For this study, four items belong there, and the first three come directly from earlier lessons.

  • The deployed task delivered a point for every click on the active circle. Reinforcement was continuous and identical for both circles, so no comparison between schedules is not possible, regardless of how the study was described.
  • Phases ran in a fixed order, so the change in responding across phases is confounded with time on task. Alternative designs might separate them better.
  • One participant's data was recorded despite a consented value of false. This is a defect in the experiment rather than in the data, and it should be reported as such and fixed before further collection.

Key points

  • The report is where an analysis becomes a claim, and it needs a validation pass of its own. Fluent prose and accurate prose are not distinguishable by reading.
  • Begin with a participant flow that reconciles: rows removed plus rows analyzed equals rows collected.
  • The methods section is a record of decisions, not observations. Exclusions, weighting, and measure definitions all change the numbers and none are visible in a results table.
  • Read every draft sentence against the computation that supports it. Cut what cannot be sourced, particularly any statistical test that was never run.
  • Keep findings and interpretations separate. Rate of responding is a finding; preference is an interpretation.
  • State what the design cannot answer. For this study, the schedule comparison and the absence of a reversal are both limitations that come from knowing the instrument.

Exercise

Draft a results paragraph for this study yourself, then read it adversarially: for each sentence, write the number that supports it in the margin. Cut any sentence with an empty margin. Then write the limitations paragraph without consulting the list above, and compare.

Checklist

Use this on any report drafted with an AI system, before it goes to a co-author. Read it yourself, and paste it to Claude to have it audit the draft.

The accounting

  • A participant flow is reported: enrolled, each exclusion with count and reason, and the number analyzed.
  • Rows removed plus rows analyzed equals rows collected.
  • The percentage of data removed is stated, not left for the reader to infer.
  • The sample size in the results prose matches the analyzed sample, not the enrolled sample.

The decisions

  • Every exclusion rule is stated with its justification.
  • The measure is defined, including any denominator (here, phase durations).
  • The weighting is stated (participants weighted equally, or observations pooled).
  • Any n that varies between cells is reported with the cell, not just once at the top.
  • The simulated or real provenance of the data is disclosed.

The claims

  • Every sentence traces to a specific computed number.
  • No statistical test is reported that was not run; no p-value appears without its test.
  • Findings and interpretations are distinguishable in the prose.
  • No claim depends on a feature of the study that the instrument did not actually implement.
  • Effects are described at the level the design supports (consistent with, rather than confirms).

The limits

  • What the design cannot answer is stated explicitly, including confounds from order or timing.
  • Defects in the experimental tasks discovered during analysis are reported and flagged to be fixed.

To use this with Claude, paste the checklist and add:

Audit this draft report against each item on the checklist. For every sentence in the
results, cite the specific number or analysis supporting it, and mark any sentence that is
an interpretation rather than a finding. Flag any claim that assumes a feature of the study
the deployed instrument did not implement. Do not improve the wording of an unsupported
sentence; identify it so I can decide whether to cut it.