Skip to content

Lesson 4 of 7 · 6 min read

Shoot in pieces

Generate one silent handheld clip per beat, cut the voice at its silences, lip-sync each clip, pin b-roll frames, and check every clip on a frame grid.

A moment at work · fictional example

Jo has one start frame and one 47-second voice take. Maya is on camera for most of the ad, and every second of video costs money. Jo has to decide how to shoot her before spending any of it.

You will make: Shoot the clips for your ad. Follow the fictional example first, then try it with material you are permitted to use.

One silent clip per talking beat

Make each talking clip with an image-to-video model, from the start frame. Generate it without sound. The voice comes from the take you chose in Lesson 3, and a lip-sync model joins the two.

Group the script into a few talking beats. Each one is a stretch the creator says without a cut. Make one clip per talking beat, as long as that beat’s slice of the voice.

A short clip gives the model less time to drift. A bad clip then costs one beat to regenerate, not the whole ad.

Write the handheld motion into the clip prompt. A phone held at arm’s length is never still. Ask for a small drift, one slight reframe partway through, and focus that shifts a little. Ask for natural head movement, with the creator talking to the camera.

Cut the voice at its silences

Each clip needs its own slice of the voice take. Cut the take where the speaker pauses, not at a time you guess from the script. The ffmpeg silencedetect filter lists every silence with its start and end times.

Emeri uses a noise threshold of -38 dB and a minimum silence of 0.18 seconds. Adjust both until the list shows the pauses between sentences and not the gaps inside words.

Pick the silence nearest each beat boundary, and cut in its middle. Each slice then starts and ends without speech, so no word is clipped.

Lip-sync each clip, then check the lips

Give a lip-sync model one clip and its slice. It changes the mouth to match the audio and keeps the rest of the clip. Do not use a talking-avatar model that generates the whole face from audio. It can change the face from your start frame.

Check lip closure on P, B and M sounds. Both lips close to make each of those sounds, so a mouth that stays open on “pantry” or “money” looks wrong.

Find those words with speech-to-text word timestamps. Crop the mouth at each one, and step through the frames.

Pin both frames for b-roll

B-roll is the short footage between the talking clips, such as hands at a fridge. Make it with the same image-to-video model.

When objects must stay put, give the model both a first frame and a last frame, if it accepts one. Make both frames with the image model, from the same scene. With only a first frame, objects can vanish or appear as the clip runs.

Check every clip on a frame grid

A frame grid is one image that shows a frame from each second of a clip, in rows. The ffmpeg tile filter makes it. Look for hands, faces and any text in the scene.

Read each grid before you lip-sync or edit. A bad frame found here costs one regeneration. Found after the edit, it costs the edit as well.

Agree the spend cap before the first paid call

Write the cap down, and get it approved before the first paid call. Estimate each call from the vendor’s price page, and add room for regenerations.

Log every call with its price. Stop and ask before any call that would pass the cap. If the vendor’s API takes an idempotency key, send one with each request. A retry with the same key returns the first job instead of charging again.

Worked example · Fernway

Jo shoots Maya’s clips

Jo has the start frame, voice take A and the spend log from Lesson 3. Jo groups the nine script beats into three talking beats. The first runs from the hook to the human beat. The second runs from the turn to the quiz question. The third runs from the reveal to the call to action. Jo plans one silent clip per beat.That answers the question you guessed at: Guess: should Jo generate one long talking clip for the whole script, or split the shoot some other way?

  1. Jo runs silencedetect on take A at -38 dB and 0.18 seconds. The list shows a pause after “not the bin” and a pause after the quiz question.
    What would you do next?Jo’s script timing puts the second boundary at 30.8 seconds, inside the last word of the question. The nearest silence runs from 31.0 to 31.3 seconds. Where should Jo cut?Think it through, then show the answer

    At 31.15 seconds, the middle of the silence. A cut at 30.8 would clip the last word. Jo cuts the first boundary the same way, and the slices run 18, 13 and 15 seconds.

  2. Jo writes one clip prompt per talking beat, each from the start frame. Each prompt asks for Maya talking to the camera with the phone at arm’s length. Each asks for a small drift, one slight reframe and focus that shifts a little. The lengths match the slices.
  3. Jo checks a frame grid of each silent clip. All three show the same face and the same sweater, and the visible hand has five fingers in every frame.
  4. Jo runs the lip-sync model on each clip with its slice. Then Jo checks lip closure at “pantry”, “bin” and “money”, using word timestamps from whisper.cpp. The lips close on each one.
  5. Jo makes three b-roll clips with the same model. They show hands lifting a bag of spinach from the back of a fridge, a quiet kitchen on a Sunday night, and a packed bag by the door. The first try at each uses only a first frame.
  6. The grid of the spinach clip shows a hand with six fingers in one frame.
    What would you do next?What should Jo do?Think it through, then show the answer

    Regenerate the clip with both frames pinned. Jo makes a last frame with the image model, from the same scene, with the hand holding the bag. The new grid shows five fingers in every frame.

  7. Jo checks the spend log: $19.80 of the $25 cap. One clip request timed out, and the retry sent the same idempotency key, so the vendor charged once.

Jo has three lip-synced talking clips, three b-roll clips, a frame grid for each, and $5.20 of the cap left. Nothing is edited yet. Lesson 5 records the real pantry check on a phone-sized screen.

Go deeper: the research and edge cases

Optional. The exercise needs only what is above.

Tuning silencedetect

The filter’s defaults are -60 dB and two seconds. Those settings suit long gaps, not the short pauses between spoken sentences.

A recording with background noise needs a higher threshold, such as -30 dB. A fast speaker needs a shorter minimum. Run the filter, read the list against the script, and adjust one setting at a time.

How often the grid samples

At one frame a second, an 18-second clip fills a grid of six columns and three rows. A fault that lasts less than a second can fall between samples. For a clip with fast hand movement, sample two frames a second and make two grids.

Starter prompts

  • Talking clip prompt, one per beat

    Animate this start frame into a [length]-second vertical video. [The creator] holds a phone at arm’s length and talks to its front camera, with natural head movement and small expressions. The camera is handheld: a small drift, one slight reframe partway through, and focus that shifts a little. Keep the same face, hair, clothes, room and light as the start frame for the whole clip. No cuts, no zoom, no text, no other people. No sound.

  • Find the silences and cut the voice (ffmpeg)

    # List every silence in the take, with start and end times in seconds.
ffmpeg -i voice.wav -af silencedetect=noise=-38dB:d=0.18 -f null - 2>&1 | grep silence_

# Cut one slice per talking clip, in the middle of the chosen silences.
# Replace 18.05 and 31.15 with your own cut points.
ffmpeg -i voice.wav -ss 0 -to 18.05 -c:a pcm_s16le slice-1.wav
ffmpeg -i voice.wav -ss 18.05 -to 31.15 -c:a pcm_s16le slice-2.wav
ffmpeg -i voice.wav -ss 31.15 -c:a pcm_s16le slice-3.wav

  • Frame grid and mouth check (ffmpeg and whisper.cpp)

    # One frame per second, tiled into one image. An 18-second clip fills 6x3.
ffmpeg -i clip-1.mp4 -vf "fps=1,scale=270:-1,tile=6x3" -frames:v 1 clip-1-grid.png

# Word timestamps for a slice, to find every word with a P, B or M sound.
./build/bin/whisper-cli -m ./models/ggml-base.en.bin -f slice-1.wav -ml 1 -oj

# Six frames from 4.2 seconds, cropped to the mouth. Set the crop box (width:height:x:y) to your frame.
ffmpeg -ss 4.2 -i clip-1-synced.mp4 -frames:v 6 -vf "crop=360:200:360:820" mouth-%02d.png

  • Plan the shots and the spend

    Here is my locked ad script: [paste the script]. Here are the cut points from silencedetect: [paste them]. Group the script into talking beats, one per slice. For each beat, write an image-to-video prompt from my start frame with handheld motion, and give its length. List the b-roll shots the script needs, and mark which need a pinned first and last frame because objects must stay put. Then estimate the cost of every call from these prices: [paste the prices from the vendor pages], with one regeneration per clip. Compare the total with my cap of [amount]. Do not assume any price I have not given you.

When not to use AI

  • Footage where the product must look exact, such as your app’s screens. Record the real product instead, as Lesson 5 does.
  • A talking-avatar model that generates the whole face from audio. It can change the face from your start frame. Use a lip-sync model on your own clip.
  • Spending past the cap because one clip is nearly right. Stop, show the frame grid, and ask whoever approved the cap.
Recall from memoryWhy does Jo make one clip per talking beat instead of one long clip?Answer it without looking back, then show the answer

A short clip has less time to drift, and a bad one costs one beat to regenerate, not the whole ad. Each clip gets its own slice of the one voice take, cut at a silence, and its own lip sync.

Exercise · your project

Shoot the clips for your ad.

  1. Write your cap down, and get it approved before the first paid call.
  2. Run silencedetect on your voice take, and cut one slice per talking beat.
  3. Generate one silent clip per beat from your start frame, with handheld motion in each prompt.
  4. Check a frame grid of every clip, then lip-sync each clip and check the P, B and M sounds.
  5. Make your b-roll, pinning both frames where objects must stay put, and log every call.

Your project workspace

Leave with work you can use.

Write here or use an approved tool. Use fictional or permitted information. Your work stays in this open page and disappears on reload. Nothing is submitted to Emeri or an AI service.

This is self-review. Completion is saved on this device and does not award certification.

Sources