Checking the work · for data analysts
Checking AI output in data analysis
A query that runs is not a query that is right. AI-written SQL can join at the wrong grain, filter out nulls without telling you, or compute an average of averages. An AI summary of a spreadsheet can state a total that no cell supports. The result looks finished, so the error survives until a stakeholder finds it.
Checking the work in analysis means reconciling. You compare totals against a trusted source, test the query on a case where you know the answer, and read the logic line by line. You also check the claims in the narrative, especially any that imply cause.
Where it comes up
Reconciling a new query
Compare the query’s total against a dashboard or report you already trust for the same period. If the two differ, find out why before using either.
Verifying numbers in an AI summary
When AI summarizes a file, recompute each figure it quotes from the file itself. Models can state sums and percentages that no calculation produced.
Reviewing a causal claim
If a draft says a feature launch caused a rise in retention, check whether the data can support cause. A before-and-after comparison usually cannot.
What it looks like at each band
E1 · Foundational
Checks that the query runs and the output looks reasonable. Catches errors that are obvious, such as negative counts. May pass on a figure that is plausible but wrong.
E2 · Proficient
Reconciles new results against a known source and tests queries on cases with a known answer. Recomputes figures quoted in AI summaries. Removes or corrects anything that does not reconcile.
E3 · Distinguished
Checks in proportion to the stakes: an exploratory look gets a quick sanity check, a board metric gets a full reconciliation. Looks for silent filters, join fan-out, and changed definitions. Challenges causal language and asks what else could explain the pattern.
E4 · Exceptional
Sets the validation standard for the team, such as which outputs need a second reviewer and which tests run before publishing. Can explain the exact cause of a discrepancy. Catches subtle errors others miss, like a time zone shift that moves a day of revenue.
Common mistakes
Trusting a query because it ran
Syntax errors are the easy case. Logic errors return rows without complaint. Test the query on a case where you already know the answer.
Letting AI do arithmetic on a pasted table
Language models can misstate totals and percentages. Have the model write code that computes the figure, then run the code yourself.
Accepting correlation as cause
A model will describe two lines moving together as one driving the other if you let it. Ask what else changed in the same period, and say so in the write-up.
A worked example
Reviewing an AI-written query that reports a sharp rise in repeat purchase rate after a loyalty program launched.
The query runs, the chart shows the rise, and the summary says the loyalty program drove repeat purchases. Paste the chart and summary into the weekly update.
Nobody checked the query logic or reconciled the figure. The summary claims cause from a before-and-after comparison. If the query changed the customer base or the definition, the rise may not exist.
Read the query line by line. It filters to customers with a loyalty ID, which did not exist before launch, so the comparison is not like for like. Rewrite it to use the same customer base in both periods. Reconcile the base rate against the existing retention dashboard. In the update, report the change and state that other factors in the period, such as a seasonal sale, were not ruled out.
The reviewer found a filter that changed the population, fixed it, reconciled the result, and kept the claim within what the data supports. The update can now survive questions.
The full study guide for this skill, for every role, is Checking the work. The other five skills for data analysts are on the AI skills for data analysts page. A one-page summary is in the AI fact-checking checklist.
See where you stand
The free check grades checking the work and the other five skills in about fifteen minutes. Nobody sees the grade but you.
Then build the skills and prove it. The three steps for data analysts list the courses and templates that fit the work.