How MetaHuman lip-sync actually works (and why most of it looks fake)

3 August 2026 · 7 min read

I spent three weeks driving a mouth from the wrong signal. This is what I got wrong, in case you are about to make the same mistake.

The mistake: following the volume

It seems obvious. Loud sound, open mouth. Quiet sound, closed mouth. Read the amplitude of the audio, map it to how far the jaw drops, done.

What you get is a flapping jaw. Every single sound produces the same shape, only bigger or smaller. There are no vowels. Turn the sound off and you cannot read a word of it — and your brain, which reads lips far more than you realise, quietly files the whole thing as a puppet without being able to tell you why.

Almost every cheap talking avatar does this, and it is the single biggest reason they look wrong.

The fix: shape from the letters, size from the sound

The shape of a mouth comes from what is being said. The size of the movement comes from how loudly it is said. They are two different signals and they need two different sources.

So: read the letters that are about to be spoken and pick a shape for them. Read the audio envelope and use it only to decide how far that shape opens.

• a → open
• e, i, y → wide
• o, u, w → round
• m, b, p → pressed shut
• everything else → a generic consonant, slightly open

That is a crude phoneme model and it is nowhere near linguistically correct. It does not need to be. It needs to produce a mouth you can read at 30 frames a second, and it does.

Four numbers

The whole face runs on four values, thirty times a second:

1. how open the jaw is
2. how wide the lips spread
3. how round they are
4. how hard they press shut

Because those are four generic numbers rather than anything MetaHuman-specific, the same stream drives anything with blend shapes. I have run the identical output on a MetaHuman in Unreal and on a browser avatar without changing a line — the names differ, the numbers do not.

The 60-millisecond rule

This is the part that sounds wrong until you try it.

The mouth has to move about 60 ms before the sound.

Between the audio device's own buffer, the engine's frame time, and the fact that shapes take a few frames to physically open, the picture always arrives after the voice. Past roughly 125 ms of lateness, viewers stop believing the face — they cannot say why, they just know.

So you push the mouth early, and it lands on time. Sixty milliseconds is a good default; it varies with the sound card and how fast the engine is running, which is why it should be a slider and not a constant.

Opening faster than closing

A vowel in normal speech lasts two to four frames. If your shapes take five frames to reach their target, they never get there — every sound comes out as the same half-open mumble, no matter how good the phoneme model is.

So the shapes have to move faster than feels natural when you are tuning them in isolation. And they need three different rates, because a real mouth is not symmetric:

• Opening is the slowest.
• Closing is roughly twice as fast — a jaw drops slowly and shuts quickly.
• Pressing shut for a b or an m is faster still, because those consonants are abrupt by definition.

Get these wrong and no amount of realism in the model saves you.

The things that are not the mouth

Once the mouth works, what remains of the uncanny feeling is usually four things, and none of them are lip-sync:

• Blinking on a metronome. Every three seconds exactly is the one thing a living face never does. Jitter it.
• A perfectly still head. Drift it a fraction of a degree, continuously.
• A dead face between sentences. Hold a small resting expression so there is something there when the mouth stops.
• Expressions that snap. They should arrive, not appear.

One I only found by testing: my blink routine was resetting the face to neutral. She would look serious, blink, and come back cheerful. The blink now jitters around whatever mood is being held instead of overwriting it.

The test

Turn the sound off and watch.

If you can still read roughly what is being said, the mouth is working. If every sound looks the same, you are following the volume — and no amount of shader work on the skin will fix it.

This is the engine inside the SinaLink plugins. If you would rather not rebuild it, get in touch — or just take the 60 ms and the three slew rates, they are the parts that cost me the most time.