Lesson 8 of 8 · 7 min read

Check that it worked

Measure learning with scenario questions after the video and weeks later, add a three-item manager checklist, and plan how the video gets updated.

Satisfaction is not learning

Most training ends with a rating form. The form tells you whether people liked the video. It does not tell you whether they can do the task.

Sitzmann and colleagues pooled studies of trainee reactions in a 2008 meta-analysis. A 2022 research summary of that work reports that reactions related only modestly to immediate knowledge. They were unrelated to knowledge tested later.

Will Thalheimer’s Learning-Transfer Evaluation Model, known as LTEM, makes the same point as a ladder of eight tiers. In order, they are attendance, activity, learner perceptions, knowledge, decision-making competence, task competence, transfer, and effects of transfer. Attendance and completion sit at the bottom. A completion report shows who opened the video, not who learned.

Keep a rating form, if you like, for comments on sound or length. Do not treat it as evidence of learning. The four parts below measure the tiers that matter.

The four-part check

First, put the scenario questions from Lesson 2 right after each segment. They check knowledge while it is fresh. Lesson 2 explained why answering them is also practice.

Second, send a parallel set of scenario questions one to four weeks later. Third, give each manager a three-item checklist to observe the task on the job. Fourth, where a production choice is still open, run two versions against the same questions.

The delay matters. A gap of one to four weeks gives forgetting time to show, which a test right after the video cannot. Send the parallel set on a fixed date, with the same number of questions as the first set.

Philippa Hardman, a learning designer, proposes that fourth part as a three-way test. One group sees an avatar, one a human presenter and one a voice-over, each with the same visuals. She argues for judging formats by measured comprehension, not by satisfaction surveys. EdzLMS suggests about 30 learners for a two-version pilot.

Write the parallel set

A parallel question tests the same decision as its partner from Lesson 2. Change the surface details, and keep the rule the learner must apply. Use a different customer, item, date or reason.

Where a rule has a limit, move the new case near the limit. If the first question used day 28 of a 30-day window, the parallel one might use day 30 or day 31. A case near the limit shows whether the learner learned the rule or remembered the example.

Write the parallel set before release, at the same time as the first set. If you write it after you see the first results, you may make it easier without meaning to.

The manager checklist

Questions show what people know. The manager checklist shows what they do. Keep it to three items, each an action a manager can see during a normal shift. Each item gets a yes or no, with space for a note.

Write each item as an action, not a state of mind. “Checks the delivery date against the 30-day policy before choosing” can be observed. “Understands the policy” cannot. Agree a time window with each manager, such as the first two weeks after go-live.

Tell staff about the checklist before the window starts. It observes the process, not the person. Say that the results go into the video plan, not into performance reviews, and keep to that.

Read the results, then act

Look at each question, not only the total score. A question that most people miss points to one place in the script or storyboard. Go back to that place first. A new voice or an avatar will not fix a step the script never taught.

Compare the immediate and delayed scores for each question. A question that scored well right after the video and poorly weeks later points to a step people did not practice enough. A question that scored poorly both times points to the explanation itself.

Small groups need care. With 25 learners, one person is 4% of the group, so a change of one or two answers can be chance. Look for questions that many people miss, not for small shifts.

Check the manager checklist against the questions. When both point to the same step, you have two kinds of evidence for one fix.

Plan for updates

The process will change, and the video has to change with it. Update cost depends on the format. A Synthesia SCORM package updates when you replace the video. Synthesia’s KnowBe4 export needs an edit, a republish and a re-export, and then the course updates in place.

A screen recording has to be recorded again when the screen changes. For software steps that change often, a step-by-step guide may cost less to keep current. Trainn, a competing vendor, says Scribe guides update by capturing the steps again, while Guidde videos need manual edits.

Keep the locked script, the pronunciation list and the questions as your master copy. Every update starts from them, and the QA pass from Lesson 7 runs again on each new render.

Worked example · Larkfield Supply

Riley reads the results at three weeks

Riley released the video on November 17, with the four scenario questions after segment one. The parallel set went out on December 8, three weeks later. Morgan and the office manager used the checklist during the first two weeks of December.

  1. Riley writes the parallel set before release, aimed at the same three tester mistakes. The new Question 1 uses a desk chair ordered 37 days ago and delivered 29 days ago. The new Question 2 uses out-of-stock toner. In the new Question 3, a customer returns a damaged shredder without saying what they want. The new Question 4 is a return 35 days after delivery, from a customer who insists on a refund.
  2. Right after the video, 24 of the 25 staff answered the questions. Each question had at least 21 correct answers.
  3. At three weeks, 23 people took the parallel set. Three questions held up, with 20 or more correct. On the new Question 4, only 11 of 23 chose to refer the return to Morgan. Most of the rest issued the refund.That answers the question you guessed at: Right after the video, Riley’s staff answer the scenario questions well. Guess: do they still choose right on a parallel set three weeks later?
    What would you do next?Is the problem the AI voice, the questions or the script?Think it through, then show the answer

    The script. The other three questions held up with the same voice. Riley rereads the script and finds that it states the after-30-days rule in one sentence. No example on screen shows a blocked return.

  4. The manager checklist has three items. Checks the delivery date against the 30-day policy before choosing. Asks the customer to choose refund or replacement. Logs the item’s condition before it leaves the returns bench.
  5. Managers observed 18 people. The second and third items were yes for 17 and 18 people. The first was yes for 12. Two of the notes describe staff who tried to override the block on a late return.
  6. Riley adds a second worked example to segment one: a return on day 35, the block message on screen, and the referral to Morgan. Riley generates the segment again, runs the Lesson 7 QA pass, and uploads the new MP4.

One change to the script came from the data, not from a hunch. Riley schedules one more question on a late return for January, to check that the new example worked. Riley also adds the day-35 case to the one-page screenshot guide, so the job aid and the video teach the same rule. The video plan records the results, the change and the date.

Starter prompts

  • Write a parallel question set

    Here are my learning objective and my scenario questions, with answers: [paste them]. For each question, write one parallel question that tests the same decision with different surface details: a different customer, item, date or reason. Keep the difficulty the same. Where a rule has a limit, such as a deadline, put the new case near the limit. Use only facts about the process that appear in my questions or in this policy text: [paste the policy]. Mark any fact you need but do not have as [CHECK].

  • Draft a manager checklist

    Here are the steps of the task my training video teaches: [paste the steps]. Write a three-item checklist a manager can use while watching an employee do the task during a normal shift. Each item must be an action the manager can see, answered yes or no, with space for a note. Do not include items about attitude or understanding.

  • Read my results

    Here are my results: [for each question, paste how many answered right after the video and how many were correct, then the same for the delayed set; then paste the manager checklist counts and notes]. For each question, say whether the score held, fell or stayed low, and which part of the script or storyboard to recheck. Do not guess causes the numbers cannot show. Flag any result where the group is too small to read with confidence.

When not to use AI

  • Judging the video by a satisfaction survey or a completion report. Neither shows whether people can do the task.
  • Letting an assistant decide what the results mean. It can sort the numbers, but you and the managers know why a step goes wrong on the job.
Recall from memoryName the four parts of the check, and say why the parallel set comes one to four weeks later.Answer it without looking back, then show the answer

Scenario questions right after each segment, a parallel set one to four weeks later, a three-item manager checklist, and two versions tested where a choice is still open. The delay gives forgetting time to show.

Exercise · your project

Add the follow-up check to your video plan.

  1. Write one parallel question for each scenario question from Lesson 2, with at least one case near a limit.
  2. Write a three-item manager checklist, and agree an observation window with the managers.
  3. Set the dates: questions right after the video, and the parallel set one to four weeks later.
  4. Write one line on how the video gets updated when the process changes, and where the master script lives.

Sources