The limits of one-shot analysis

Concept

You have collected your data. The files are sitting in a folder, and you have an AI system that will read them. The obvious thing to do is hand over the files and ask what happened. Within a minute you will have counts, averages, a comparison across conditions, and a paragraph that interprets the result.

This lesson puts that request to Claude with our dataset and examines the response. What comes back is coherent and well organized, and it answers the question that was asked. The difficulty is that an inappropriately phrased request leaves several consequential decisions to be settled out of sight: who counts as a participant, which rows are real, what should the numbers be divided by, and what was the study actually capable of showing. Those decisions get made either way. If you do not make them, you inherit whatever Claude decided without knowing what they were.

Everything that follows in this half of the course is the process of making those decisions deliberately.

A note on the data

The dataset here is simulated. We ran the real task on a few pilot participants, whose data is kept separately in data/pilot/, but three people are too few to demonstrate a group-level analysis. So we generated a cohort of 40 participants (data/synthetic/) whose behavior was built to match the real task. We did this so the workshop can show the single-subject analysis common in behavior analysis and the group-level analysis other fields expect. We also seeded it with the kinds of quality problems real online data contains, so there are genuine challenges to practice catching. Everything here is simulated unless we say otherwise.

The one-shot request

This is the prompt most people would write, and there is nothing wrong with it as a question. As an instruction it is incomplete.

Ask Claude:

Here are three CSV files from my experiment: participants.csv, demographics.csv, and
responses.csv in data/synthetic/. Analyze the data and tell me what happened.

What comes back:

Sample:      57 participants, 11,799 responses
Demographics: mean age 36.9, 10 gender categories reported
 
Responses by phase:   phase 1: 5,450    phase 2: 5,471    phase 3: 878
Responses by button:  R1: 5,231   R2: 5,282   distractor: 1,286
Reinforced:           75.8% of all clicks
Mean responses per participant: 207
 
Mean response rate (clicks per minute)
                phase 1    phase 2    phase 3
R1                 48.4        6.8        7.1
R2                  7.1       48.8        6.5
distractor          5.7        5.5       10.8
 
Interpretation: responding tracked the reinforced button across phases, and dropped
sharply in phase 3, an 84% decline from phase 1.

Reading the output: In phase 1 participants clicked the R1 circle about 48 times a minute and the other two circles about 6 times a minute, and in phase 2 the pattern moved to R2. The closing sentence says responding fell by 84 percent once no clicking of any circles led to points.

This is a reasonable-looking analysis. The structure is right, the interpretation matches the design of the study. And the central claim that clicking followed the points, is one we will still be making at the end of the course. No part of the output identifies a problem, and there is no error message anywhere in it. Four separate figures in that summary are nonetheless wrong, and the most consequential problem of all doesn't even appear in that output.

The sample is not 57 people

The study enrolled 40 participants. The summary reports 57 because it counted the distinct participant identifiers appearing in the response file rather than the people in the participant file. Eighteen of those identifiers appear on exactly one response row each and match no participant in the study. Of the forty real participants, one produced no responses at all, and one is not a person clicking, for reasons given below.

The correct figure for analysis is 38. Everything in the summary that is expressed "per participant" is therefore divided by the wrong number, including the mean of 207 responses per participant, which is 240 once the sample is what it should be. We retutn to this in lessons 13 and 14.

The 84% decline is mostly the clock. Phases 1 and 2 each lasted two minutes. Phase 3

lasted one minute. Comparing 5,450 clicks in phase 1 with 878 clicks in phase 3 compares two periods of different length, so part of the difference is only that there was half as much time to click.

Correcting for duration alone, responding went from about 2,725 clicks per minute across the group in phase 1 to about 878 in phase 3. This is a decline of 68% rather than 84%. The decline is still real and it still a primary finding of the study. But the number reported is not an accurate measurement of that main finding. Comparing periods of unequal length is the subject of lesson 16.

Nearly a quarter of the data came from one responder

Of the 11,799 response rows, 2,662 belong to a single participant whose median gap between clicks is 97 milliseconds (~10 responses per sec). A person cannot click at that speed for five minutes. That one identifier contributed 22.6% of every row in the file, and so it contributed to every average in the summary.

Removing that likely bot responder along with the eighteen unattributable rows, removes 22.7% of the file and changes the table with the main findings:

                one-shot     after exclusions
phase 1  R1        48.4                 43.6
phase 2  R2        48.8                 43.9

The direction of the finding does not change. But, the magnitude does, by about ten percent, and it changes because of one source of rows that no one decided to include. Deciding who belongs in the analysis is the subject of lesson 14.

The demographics describe values that are not what they appear to be

The reported mean age of 36.9 includes a participant recorded as 142 years old. With that value treated as missing, the mean is 34.2. The ten gender categories are ten spellings of about four intended answers, including Male, male, and Man as three separate rows in the count. Neither of these produced an error, because neither is malformed as data. A number can be in range for its column and still be impossible for a person.

The error that data checks may not catch

The four problems above are all visible in the data. Historically, the researcher familiar with the experiment would've been hands on with the data and likely would have caught these oddities. But, by simply uploading data into AI systems, that familairity is lost. This doesn't mean you shouldn't use AI for these kinds of analyses. But it does mean that you need to know how to prevent, identify, and remove these kinds of errors which researchers and data scientists used to have to do manually. The rest of this course will teach you to find them. The fifth problem is not visible in the data at all.

The written description of this study, in lesson 8 of the build half, says the two circles led to points on different reinforcement schedules. The task that was actually deployed led to a point for every click on whichever circle was currently active. That is, reinforcement was continuous, and identical for both circles.

If you supply the study description along with the data, the analysis that comes back will include a comparison of the two schedules and an interpretation of the difference. The comparison will be arithmetically correct but scientifically inaccurate because the thing actually compared does not match the description. No amount of checking the data catches this, because the data are fine. The only thing that catches it is checking what the experiment actually does, by reading or investigating the code itself rather than the document that describes it.

This is why the analytic process is worth understanding thoroughly rather than delegating without question to an AI system. How to handle those items above (and others) involves making assumptions and decisions that influence the analytic result. Those assumptions and decisions don't disappear when they are automated. They are made somewhere else, out of sight, and they may not be the ones you would have made.

Key points

  • A one-shot request produces an answer that has not been verified or validated. Verified and validated answers look the same as an unverified and invalid one. There is no error message.
  • Every consequential decision left unspecified in the prompt is made anyway, outside your view (e.g. whom to count, what to exclude, what to divide by, how to handle missing data).
  • In this dataset, the one-shot summary got the sample size, the per-participant mean, the magnitude of the phase-3 decline, the group rates, and the demographic summaries wrong. All the while, it reached a defensible overall conclusion that looks right.
  • The most consequential error came from outside the data. There was a mismatch between what the study was described as doing and what the deployed task did.
  • Confirm what your experiment did by examining the preparation, not the write-up.

Exercise

Run the simple one-shot prompt above on data/synthetic/ yourself and keep the output. You will not be able to evaluate most of it yet. Write down three things the response asserts that you can't currently verify, and keep the list. At the end of the course, return to it and check whether you could verify them now.

Checklist

Use this before accepting any analysis you did not carry out step by step. Read it yourself to see what a one-shot answer left unstated, and paste it to Claude to have it audit its own output. A ready prompt is at the bottom.

  • The number of participants in the analysis is stated, and matches the number enrolled minus documented exclusions.
  • It is clear what one row of each file represents, and any per-participant figure is divided by participants rather than rows.
  • Any comparison across periods of different length is expressed as a rate, not a raw count.
  • Every exclusion is named and justified, and the total proportion of data removed is reported.
  • Summary numbers are compared with what is physically possible for a person (e.g., response speed, age, session duration).
  • The analysis is confirmed by checking what the instrument actually did, rather than the study description.
  • Any claim the design cannot support is stated as unsupported rather than omitted.

To use this with Claude, paste the checklist and add:

You have just given me an analysis of the data in data/synthetic/. Audit your own output
against each item on this checklist. For each item, state what you assumed, whether you
verified it, and what would change in your reported numbers if the assumption were wrong.
Where you cannot verify an item from the data alone, say so explicitly rather than
inferring it from the study description.