Prime Video tests AI that matches lips to dubbed English audio

In short: Amazon Prime Video is rolling out a feature that uses AI to make actors’ lips look like they match dubbed English speech.

What happened

Amazon says Prime Video now has a new AI-powered lip sync feature for dubbed shows. Lip sync means the actor’s mouth movements line up with the words you hear, like a movie that looks properly “spoken” instead of slightly off.

For now, the feature is only available on the English dub of the German series Maxton Hall. An English dub is a version where the original audio is replaced with English-speaking voice actors.

Prime Video says it uses a mix of AI and visual effects to adjust the video so the mouth movements better match the translated dialogue. It is similar in idea to editing a video so it looks like the person is saying different words, except Prime Video says it is designed to work with the dubbed audio track.

Amazon has been exploring this area already. Last year, Prime Video experimented with “AI-aided” dubbing in English and Spanish across 12 movies and TV shows.

The company says the new lip-sync dubbing is available for seasons 1 and 2 of Maxton Hall for subscribers worldwide. It will also be used for the show’s third season, which Prime Video says will launch on December 9.

Why it matters

Dubbed shows can be easier to watch than subtitles, but mismatched mouth movements can be distracting. If this works well, it could make foreign-language shows feel more natural in other languages, and it could encourage streaming services to offer more dubs across more titles.

Source: The Verge AI