Lesson 2 of 8 · 7 min read
Write the check first
Write one learning objective and three to five scenario questions before any script, so the video teaches what the check asks.
Questions before the script
The usual order is script, then video, then a quiz added at the end. This lesson reverses it. You write the learning objective, then 3 to 5 scenario questions that show whether someone met it. Only then do you write the script, in Lesson 3.
The questions do two jobs. They decide what the script must cover, because anything a question needs has to be taught. They also become the check at the end of each segment, and Lesson 8 repeats a parallel set weeks later.
A script written first tends to cover what is easy to say. A script written after the questions covers what the learner must do. That difference shows up in every later lesson.
Start from one objective
A learning objective names what the learner will do, under what conditions, and to what standard. “Understand the returns process” is not an objective, because nobody can watch someone understand. “Process a customer return without help” is closer, because you can watch someone do it.
Keep one objective per video. If you find yourself writing two, you may have two videos. Lesson 4 splits a video into segments, but each segment still serves the same objective.
Questions help people learn, not only show what they learned
Answering questions is a way to learn, not only a way to check. Researchers call this retrieval practice. A 2017 meta-analysis by Adesope, Trevisan and Sundararajan found that practice tests beat other study conditions, with an overall effect of g = 0.61.
The same review found benefits for transfer as well as retention, and feedback made them stronger. Multiple-choice questions worked at least as well as questions where learners recall the answer unaided. That matters for a training video, because multiple choice is what most learning platforms support.
Emeri read the Learning Scientists’ summary of the paper, not the paper itself. Sources disagree on how many articles it covered. The direction of the finding is consistent across those sources.
A second line of evidence points the same way. Koedinger and colleagues estimated that more practice activity in an online course gave more than six times the benefit of the same increase in watching or reading. The data was correlational and came from one course, so treat it as support, not proof.
Write scenario questions, not recall questions
A scenario question puts the learner in a situation from the job and asks what they would do. A recall question asks them to repeat a fact, such as “How many days is the return window?” Recall questions are easy to write and easy to pass. They do not show whether someone can apply the rule to a real case.
A scenario question names a customer, an item, the dates and what the customer wants. Then it asks for the decision. The learner has to find the relevant facts in the situation and apply the rule to them, which is what the job asks.
Put every fact in the question
Each scenario question must contain every fact the right answer depends on. If the answer depends on the delivery date, the question gives the delivery date. If it depends on stock, the question says whether the item is in stock. A learner who knows the process should be able to answer from the question alone.
AI-written scenarios often break this rule. A commenter on Hacker News described a business simulator that marked an answer wrong. The expected answer depended on information the scenario never gave. The question tested a guess, not the skill.
Read each draft question and ask two things. What facts does the right answer need? Is each one on the page? If a fact is missing, add it or rewrite the question.
Draw wrong answers from real mistakes
The wrong options in a multiple-choice question are called distractors. Good distractors are mistakes people make. If staff often count the 30 days from the order date instead of the delivery date, that becomes a wrong option. A learner who picks it shows you what the video failed to teach.
Invented distractors are often easy to rule out, so the question tests less than it seems. Collect real mistakes before you write. Ask the people who handle the process now, or the people who tested the new system, what went wrong. An assistant can draft distractors, but it cannot know your team’s mistakes.
Write feedback for every option. The retrieval review found that feedback strengthened the benefit. One line that says why the right answer is right, and which rule a wrong answer breaks, turns each question into a short lesson of its own.
Worked example · Larkfield Supply
Riley writes the check for the returns video
Riley starts from the Lesson 1 decision: two segments of about 4 minutes, narrated with an AI voice. Larkfield’s returns policy has four rules. The 30 days count from the delivery date. Within 30 days, the customer chooses refund or replacement. If the replacement is out of stock, staff offer a refund. After 30 days, the system blocks the return and staff refer it to Morgan.
- Riley writes the objective: “Process a customer return in the order system, including choosing refund or replacement under the 30-day policy, without help.”
- Riley asks the two staff who tested the new screens what went wrong. They name three mistakes: counting from the order date, offering a replacement that is out of stock, and choosing for the customer instead of asking.
What would you do next?What should Riley do with those three mistakes?Think it through, then show the answer
Turn them into wrong options. Each one becomes a distractor in a question where that mistake is tempting. A learner who picks it shows Riley which part of the video needs work.
- Question 1: a customer returns an Aubriot desk lamp, product code LKS-20417. It was ordered on December 2 and delivered on December 9. Today is January 6, the lamp is in stock, and the customer wants a replacement. The right answer is to process the replacement, because it is day 28 after delivery. The distractor refuses the return, counting from the order date.
- Question 2: a box of printer paper was delivered on December 3, and today is December 15. The customer wants a replacement, and the paper is out of stock. The right answer is to offer a refund. The distractor processes the replacement and leaves the warehouse to find stock.
- Question 3: a stapler delivered 10 days ago is the wrong model. The customer asks what Larkfield can do. The right answer is to ask whether the customer wants a refund or a replacement. The distractor sends a replacement without asking.
- Question 4 began as a draft: “A customer returns an item after the window closed and insists on a refund.”
What would you do next?What is missing from the draft of Question 4?Think it through, then show the answer
The dates. A learner cannot check the window without the delivery date and today’s date. Riley adds both: delivered on December 1, returned on January 11. The right answer is to refer the return to Morgan, and the distractor issues a refund anyway.
Riley has four scenario questions. All four test decisions that segment one teaches, so all four follow segment one. Each one names the item, the dates, the stock level where it matters, and what the customer wants. Each wrong option is a mistake a tester made. Lesson 3 writes a script that teaches what these questions need, and Lesson 8 writes a parallel set for three weeks later.That answers the question you guessed at: Guess: should you write a training video’s questions before the script, or add them once the video is done?
Starter prompts
Objective to scenario questions
Here is the learning objective for a training video: [paste your objective]. Here is the process it teaches: [paste the process document or the steps]. Here are mistakes people make with this process: [list real mistakes]. Write [3 to 5] multiple-choice scenario questions. Each question describes a realistic situation from the job and asks what the learner should do. Each question must contain every fact the right answer depends on, such as dates, amounts and stock levels. Use the mistakes I listed as the wrong options, and say which mistake each wrong option represents. Use only rules from the process document. If a rule is unclear or missing, write "GAP:" and describe what is missing instead of guessing.
Check my questions
Review these scenario questions and their answers: [paste the questions]. For each question, list every fact the right answer depends on, and say whether the question gives that fact. Flag any question a careful learner could get wrong because information is missing. Flag any question that tests recall of a fact instead of a decision. Flag any wrong option that no real person would choose. Do not rewrite the questions. List the problems.
Make the objective observable
Here is a draft learning objective: [paste it]. Rewrite it so it names what the learner will do, under what conditions, and to what standard. Keep it to one sentence. If the draft holds two objectives, say so and suggest how to split it. Do not add anything the draft does not imply.
When not to use AI
- Collecting the real mistakes. An assistant can draft plausible wrong answers, but only the people who do the work know which mistakes happen.
- Settling the rule a question tests. If the process document is unclear, ask the process owner before any question depends on it.
Recall from memoryWhat two jobs do the scenario questions do when you write them before the script?Answer it without looking back, then show the answer
They decide what the script must cover, because anything a question needs has to be taught. They also become the check at the end of each segment.
Exercise · your project
Write the objective and the scenario questions for your video.
- Write one learning objective for the process you chose in Lesson 1.
- Ask two people who do the process, or who tested it, which mistakes they see.
- Write 3 to 5 scenario questions, with the first prompt below or by hand.
- For each question, list the facts the right answer needs, and check that each one is in the question.
- Add the objective and the questions to your video plan.