MMAudio: Sound for the Video You Already Made

Play your latest AI clip with the sound on. Nothing. A horse gallops across a field without a single hoofbeat, a wave breaks in total silence, a glass shatters like it's happening in a vacuum. The picture is finished and the shot still isn't, because a silent clip doesn't feel real to anyone watching it. Plenty of video generators still hand you exactly that: frames, no sound.

MMAudio exists for that gap. You give it the video you already made, add a short text prompt if you want to steer it, and it generates audio that follows what happens on screen, hoof by hoof and splash by splash. It doesn't make you a new clip. It finishes the one you have. And for most short-form work, that's the more useful job.

What MMAudio Actually Is

MMAudio is a video-to-audio model that also works as a text-to-audio model. Its authors sum it up in one line: it "generates synchronized audio given video and/or text inputs." It comes from the CVPR 2025 paper "MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis" by Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing and Yuki Mitsufuji, a team from the University of Illinois Urbana-Champaign, Sony AI and Sony Group Corporation. Cheng did the work during an internship at Sony AI.

The model is small by today's standards, and that's a feature. The paper describes four sizes: two small versions at 157 million parameters (one producing 16 kHz audio, one 44.1 kHz), a medium one at 621 million, and a large one at 1.03 billion, both of those at 44.1 kHz. That's a model built to be fast and focused, not a giant that does everything slowly.

On each::labs it runs as a video-in, video-out step. You upload a clip, and you get back an MP4 with the generated sound already laid under the picture. No separate muxing step, no lining up a WAV against a timeline by hand.

How MMAudio Keeps Sound on the Beat

Generating a plausible sound is the easy half. Any decent audio model can make "rain" sound like rain. The hard half is timing. If a door slams on frame 40 and the slam lands on frame 52, the viewer feels it immediately, even when they can't say what's wrong. Off-sync Foley is worse than no Foley at all.

MMAudio attacks timing directly. The repo describes "a synchronization module" that "aligns the generated audio with the video frames," and the paper calls it a conditional synchronization module working at the frame level. Under the hood, the model reads the video twice at different speeds: visual features at 8 fps to understand what's in the scene, and synchronization features at 24 fps to track when things happen. The slow read tells it you're looking at a dog. The fast read tells it exactly when the dog barks.

The numbers back this up. The paper measures sync with a metric called DeSync, where lower means the audio drifts less from the picture. The largest MMAudio scored 0.442 and the smallest 0.483, while the best prior public model in the paper's main comparison table scored 0.654. The authors also ran a version with the synchronization features removed, and its DeSync jumped to 0.973. In other words, the sync module is doing real work, not decoration.

Speed is the other surprise. The abstract reports "a low inference time (1.23s to generate an 8s clip)" for the small model, and even the 1.03 billion parameter version came in at 1.96 seconds in the paper's own measurements. Audio stops being the slow part of your pipeline.

Why Joint Training Is the Whole Trick

Here's the problem every video-to-audio model runs into: there just isn't much good video with clean, matching sound to learn from. MMAudio's answer is the "multimodal joint training" in its title. Instead of learning only from audio-visual clips, it trains on video and text-audio data together, in one network.

The video side comes from VGGSound, which the paper describes as around 500 hours of footage. The text side is much bigger: AudioCaps, Clotho and WavCaps, adding up to 951,000 audio-text pairs, with WavCaps alone at roughly 7,600 hours. The authors say the larger text-audio data improves semantic alignment. Put simply, the model learns what thousands of things sound like from captions, then learns when they happen from video.

That's also why the text-to-audio mode isn't an afterthought. A model that learned sound from captions can generate from a caption alone, and the paper reports competitive text-to-audio results. On each::labs the video input is required, so treat MMAudio as a Foley tool first. That's the job it was tuned for anyway.

Eight Seconds, by Design

MMAudio learned on short clips. The paper uses the first 8 seconds of each training video, and the repo puts it plainly: "The default output (and training) duration is 8 seconds." Longer or shorter durations can work, but the repo also warns that a large deviation from the training duration may lower quality.

Don't read that as a problem. Read it as a production rule. Many image-to-video clips run somewhere in the 5 to 10 second range, which sits right around where MMAudio is most comfortable. If you're building a 30-second spot, score it shot by shot, the way a Foley artist would, instead of feeding the model one long edit and hoping it stays focused. Short shots also make it easier to rerun one bad cue without touching the rest.

The each::labs page exposes a Duration field, so you can match the audio length to your clip. Set it to the clip's real length and stay close to 8 seconds when you can.

MMAudio on each::labs: The Inputs That Matter

The MMAudio model page keeps things short. There's one required input and a handful of dials.

Video is the only required field, and it's the main signal. The model reads motion, contact and scene content from the frames, so a clip with clear physical action (feet on gravel, a pan on a stove, waves on rocks) gives it far more to work with than a slow drift over a static landscape.

Prompt is optional and steers what kind of sound you get. The page's own example is a single word, "galloping." Negative Prompt tells the model what to keep out, and the example there is "music." Then there's Duration, Number of Steps, CFG Strength and Seed. The page notes that "Higher num_steps improves quality but increases processing time," so raise it for final renders and keep it modest while you're testing. CFG Strength is the familiar guidance dial for how closely the output follows your prompt, and Seed lets you reproduce a take you liked.

One practical note from the repo: the model sometimes adds background music or makes mumbled, speech-like sounds nobody asked for. That's exactly why the negative prompt field matters. Keep "music" in there for pure Foley, and add "speech, talking" when the scene has people but no dialogue.

Write Prompts for the Sound, Not the Picture

The video already tells MMAudio what's on screen. Your prompt shouldn't describe the shot again. It should describe what you want to hear: the sound source, the surface, the space and the mood. Short beats long. A long cinematic paragraph about lighting gives the model nothing it can use.

For a horse running across a field, name the contact and the setting:

Prompt: galloping horse hooves on dry earth, open field, light wind, distant birds
Negative prompt: music, speech

For a product shot of coffee being poured, go close and tactile:

Prompt: hot coffee pouring into a ceramic cup, close-up, quiet kitchen room tone, soft clink of a spoon
Negative prompt: music, talking, crowd

For a moody city scene, lean on ambience instead of single events:

Prompt: steady rain on pavement at night, cars passing on wet asphalt, distant thunder, neon buzz
Negative prompt: music, voices

And for a nature clip where you want the scene to breathe:

Prompt: ocean waves crashing on rocks, seagulls overhead, strong coastal wind
Negative prompt: music, speech, engine noise

Notice the pattern. Concrete nouns, physical surfaces, a sense of the room. The repo notes the model handles familiar concepts better than unusual ones (it can do "gunfires" but struggles with "RPG firing"), so describe what you want in everyday sound words rather than niche jargon. "Heavy wooden door slamming" will beat a technical name every time.

From Silent Clip to Finished Shot on each::labs

The workflow is two steps, and both live on each::labs. First make the picture. Pick an image-to-video model such as Kling 4.0 Image to Video, animate your still, and focus your prompt on motion. If you want help there, our guide on writing image to video prompts covers it.

Then pass the resulting clip to MMAudio with a sound prompt and a negative prompt. You get an MP4 back with audio already under the picture, ready to post or drop into an edit. Because the output is a finished video, you can chain the two models in one of the each::labs flows and stop thinking about audio as a separate errand.

Here's the part people skip: plan the motion for the sound. A model can only sync to events it can see. If you want a satisfying crunch, generate a shot where a boot actually hits the snow. If you want a door slam, make sure the door closes on camera. Clear on-screen action is the best sound prompt you'll ever write.

MMAudio vs Sonilo Video-to-SFX: Which One for Which Clip

MMAudio isn't the only video-aware sound model on each::labs. Sonilo Video-to-SFX also "generates synchronized sound effects from video scenes," and the two take different routes.

Sonilo Video-to-SFX returns an audio file rather than a finished video, and it accepts optional JSON timeline segments, so you can direct which sound happens when. That makes it a strong pick when you're mixing in an editor and want the effects as their own track, or when you need precise cue-by-cue control. Sonilo also has a Video-to-Music model for scoring, which MMAudio doesn't try to do.

MMAudio is the faster path to a finished shot. One clip in, one MP4 out, steered with a plain prompt and a negative prompt. For social cuts, product loops and quick previews, that's usually all you need. A good rule: start with MMAudio for ambience and Foley on short clips, and reach for Sonilo when you need separate stems or music. Our post on adding music and sound effects to AI video walks through the Sonilo side in depth.

Who Gets the Most Out of It

The best fits share one trait: there's a clear physical event on screen and nobody wants to spend an afternoon in a sound library. Social teams turning silent AI clips into posts that work with sound on. Ecommerce teams adding pours, clicks and fabric rustle to product loops. Game and app prototypers who want a scene to feel alive before a sound designer touches it. Filmmakers using AI shots in an animatic and wanting temp Foley that actually lines up.

It's also a good habit for anyone browsing the wider each::labs model catalog. Pick a video model for the look you want, then let MMAudio handle the sound. You don't need a video model that does both well. You need two models that each do their job.

Frequently Asked Questions

Is MM Audio the same as MMAudio?

Yes. People search for it both ways, and each::labs lists it as "MM Audio," but it's the same model the paper calls MMAudio. The name fits its multimodal joint training, which mixes video and text-audio data in one network.

Who made MMAudio?

A research team from the University of Illinois Urbana-Champaign, Sony AI and Sony Group Corporation with Ho Kei Cheng as first author. The work was published at CVPR 2025, and the code lives in the hkchengrex/MMAudio repository.

How long can an MMAudio clip be?

The model was trained on 8-second clips, and that's its default. Other lengths can work, but quality may drop the further you move from 8 seconds, so for longer edits generate sound shot by shot and assemble them.

How do I stop MMAudio from adding music?

Use the negative prompt. Put "music" there for pure Foley, and add "speech" or "talking" if mumbled voices creep into scenes with people. A specific sound prompt helps too, because it gives the model a clear target.