MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

Motivation

Ever faced the problem of having a video but no audio? Generating realistic, synchronized audio from video is a challenging task that has significant implications in various fields, including film production, gaming, and virtual reality. The ability to synthesize high-quality audio that matches the visual content can enhance user experience and immersion. Existing methods either rely on limited video-audio datasets or bolt "control modules" onto text-to-audio models, which can lead to suboptimal results. This is where the concept of multimodal joint training comes into play.

The Innovation: MMAudio proposes a novel approach for Foley i.e. to video-to-audio synthesis by leveraging multimodal joint training. By integrating video, text and audio data into a single training framework, the model learns to generate high-quality audio that is not only synchronized with the visual content but also semantically relevant. This approach allows for better generalization across different domains and improves the overall quality of the synthesized audio.

MMAudio performing multimodal joint training with high-quality, abundant audio-text data which enables effective data scaling. At inference, MMAudio generates conditions-aligned audio with video and/or text guidance.

Relevance Today:

• With the rise of content creation platforms and the increasing demand for high-quality audio-visual content, the need for efficient video-to-audio synthesis methods has never been greater.
• Multimodal learning is a hot topic in AI research, and MMAudio represents a significant step forward in this field.
• The ability to generate realistic audio from video can revolutionize industries such as gaming, film, and virtual reality.

The Evolution of Multimodal Learning for Audio-Visual Synthesis

Multimodal learning has evolved significantly over the past few years, driven by advancements in deep learning and the availability of large-scale datasets. Early approaches focused on simple feature concatenation or shallow fusion techniques, which often failed to capture the complex relationships between different modalities.

Semantic Alignment has been a foundational concept in multimodal learning, focusing on establishing meaningful connections between different modalities at a conceptual level:

• Early approaches relied on paired audio-visual data trained with either generative objectives ([2, 6]) or contrastive objectives ([42]), teaching models to understand cross-modal relationships.
• The field advanced by incorporating audio-text paired training alongside audio-visual training, creating more robust semantic understanding that could transfer between modalities.
• Innovations like ImageBind [11] and LanguageBind [30] demonstrated how joint training could create shared semantic spaces where conceptually similar content clusters together regardless of original modality.
• Modern approaches now focus on omnidirectional feature sharing, allowing information to flow freely between all modalities rather than in predetermined directions.

Temporal Alignment represents another critical dimension in multimodal learning, particularly for time-based media like video and audio:

• Traditional methods relied on handcrafted proxy features such as audio onsets [32, 33], energy patterns [17, 20], or root-mean-square of waveforms [4, 34] to synchronize modalities.
• Contemporary approaches have shifted toward learning alignment directly from deep feature embeddings, with models like Synchformer [19] enabling more nuanced interpretation of temporal relationships.
• Recent innovations in positional embeddings have significantly improved synchronization capabilities - some approaches scale up low-frequency (visual) embeddings while others subsample high-frequency (audio) embeddings to achieve better alignment.
• Higher frame-rate processing (evolving from 8 FPS to 24 FPS) has dramatically improved audio-visual synchronization quality, particularly for precise timing events.

Multimodal Conditionding approaches have transformed how models integrate information from multiple sources:

• The conventional approach involved adding "control modules" to inject visual features into pretrained text-to-audio networks [13, 17, 20, 34], but this increased parameter counts and created architectural inefficiencies.
• This modular approach faced limitations as the text modality remained fixed during video-to-audio training, forcing the video modality to adapt to text semantics rather than allowing mutual adaptation.
• Modern architectures now train all modalities simultaneously in joint frameworks, creating more coherent semantic spaces and enabling more natural cross-modal generation.
• Alternative techniques like Seeing-and-Hearing [35] attempted alignment without joint training by performing gradient ascent on alignment scores at test time, but these approaches often resulted in lower quality and temporal misalignment compared to jointly-trained models.

This historical trajectory shows a clear progression from isolated single-modality approaches toward more integrated systems that can understand and generate content across the full spectrum of human sensory experience, with each advancement building upon previous innovations to create increasingly sophisticated and capable models.

Key Learnings

Through my exploration of multimodal learning on video-audio synthesis, I've gained several important insights:

• Problem of Foley Sound Synthesis: Creating convincing, high-quality ambient sound effects for video (Foley) is complex. It requires not only generating realistic audio but ensuring it is semantically aligned and temporally aligned.

• Multimodal Joint Training Approach: A central learing is the effectiveness of MMAudio's novel multimodal joint training framework. Instead of just training on video-audio data or adapting pre-trained models, MMAudio trains on video, audio and text jointly in a single network from scratch.

• Attention Mechanisms: Cross-attention layers allow models to focus on relevant parts of one modality while processing information from another, enabling more nuanced understanding across modalities.

• Conditional Flow Matching: Conditional Flow Matching is used for generative modeling. For more details, readers can refer to [63]. For a quick overview, at test time, to generate a sample, we randomly draw noise $x_0 \sim \mathcal{N}(0,I)$ and use an Ordinary Differential Equation(ODE) solver to numerically integrate from time $t = 0$ to time $t = 1$ following a learned time-dependent conditional velocity vector field $v_theta(t, \mathbf{C}, x):[0,1] \times \mathbb{R}^C \times \mathbb{R}^d \to \mathbb{R}^d$, where $t$ is the timestep, $\mathbf{C}$ is the condition (e.g., video and text), and $x$ is a point in the vector field. We represent the velocity vector field via a deep net parameterized by $\theta$.

  At training time, we find $\theta$ by considering the conditional flow matching objective

  $\mathbb{E}_{t,q(x_0),q(x_1,\mathbf{C})}||v_\theta(t, \mathbf{C}, x_t) - u(x_t|x_0, x_1)||^2$,

  where $t \in [0, 1]$, $q(x_0)$ is the standard normal distribution, and $q(x_1, \mathbf{C})$ samples from the training data. Further,

  $x_t = tx_1 + (1-t)x_0$

  defines a linear interpolation path between noise and data, and

  $u(x_t|x_0, x_1) = x_1 - x_0$

  denotes its corresponding flow velocity at $x_t$.

• Audio Encoding: For computational efficiency, we model generative process in a latent space. For this, we first transform audio waveforms via Short-Time Fourier Transform (STFT) and extract mel spectrograms, which are then encoded by a pretrained VAE into latents $x_1$. During testing, the generated latents are decoded by the VAE into spectrograms which are then vocoded by a pretrained vocoder into audio waveforms.

• Overview: The below figure illustrates the network architecture. MMAudio consists of a series of ($N_1$) multimodal transformer blocks with visual/text/audio branches, followed by a series of ($N_2$) audio-only transformer blocks. For scynchrony, they devised a conditional synchronization module that extracts and integrates into the generation process for temporal alignment.

• Multimodal Transformer: Our desire is to model the interactions between video, audio and text modalities. For this purpose, we largely adopt the MM-DiT block design from SD3[8] and introduce two new components for temporal alignment: aligned RoPE positional embeddings for aligning sequences of different frame rates and 1D convolutional MLPs for capturing local temporal structure. Note, they also included a self-attention. This design help us to build a deeper network with the same parameter count and compute without sacrificing multimodality.

• Representations: All the features are represented in one-dimensional tokens. Absolute positional encoding is not used which allows us to generalize for different duration of videos. The visual features $F_v$ (one token per frame, at 8 fps) and text features $F_t$ (77 tokens) are extracted from CLIP as 1024d features. The audio latents $x$ are in the VAE latent space at 31.25 fps as 20d latents by default. The synchronization features $F_{syn}$ are extracted with Synchformer at 24 fps as 768d features. Except text tokens, all other follow the same temporal ordering.

• Joint Attention: These tokens from different modalities communicate via joint attention. We use the query, key and value representations from all modalities and apply scaled dot product attention. They noticed that joint attention alone does not capture temporal alignment.

• Aligned RoPE position embedding: For audio-visual synchrony, precise temporal alignment is crucial. They applied RoPE embeddings on the queries and keys in the visual and audio streams before joint attention. It is not applied to text stream. For aligning the fps, they scaled the frequencies of positional embeddings in the visual stream proportionally. They also introduced additional synchronization module for better synchrony.

• ConvMLP: For better capturing of local temperature structure, ConvMLPs are used for streams. They used 1D convolutions (kernel size = 3 and padding = 1).

• Global Conditionding: This injects global features into the network through scales and biases ind adaptive layer normalization layers (adaLN). First, they computed a global conditioning vector $c_g \in \mathbb{R}^{1 \times h}$ shared across all transformer blocks from Fourier encoding of flow timestep. Each layer modulates its input $y \in \mathbb{R}^(L \times h)$ ($L$ is the sequence length) as:

  $$\text{adaLN}_y(y, c_y) = \text{LayerNorm}(y) \cdot \mathbf{1}\mathbf{W}_{\gamma}(c_g) + \mathbf{1}\mathbf{W}_{\beta}(c_g)$$

  Here, $\mathbf{W}_{\gamma}, \mathbf{W}_{\beta}$ are MLPs, and 1 is a $L \times 1$ all-ones matrix.

• Effective Data Scaling: Joint training enables effective data scaling. This is a crucial insight for training as the datasets alone is limited by data scarcity and expense. By jointly training on larger-scale, readily available audio-text data (like WavCaps, which is much bigger than VGGSound) effectively scaling the data. This approach helps learn a unified semantic space across modalities

• Conditional synchronization Module: Token-level conditioning is developed to improve audio-visual synchrony. The cross-modality attention layers aggregate features via a soft distribution, which tampers precision. To deal with this, high frame rate (24fps) are extraced ($F_{syn}$) from input video.

• Single-Modality Performance: Surprisingly, MMAudio's multimodal approach also achieves competitive performance in text-to-audio generation, even comparable to models specifically designed for that task. So, we indeed do have rich semantic feature space.

• Addressing Data Overlaping Issues: The existence of data overlaps between common datasets used in video-to-audio research and states they have carefully removed overlapping samples from the training data for fair evaluation. This is important for unbiased results.

• Limitations: There are some limitations to the model. It generates unintelligible mumbles when prompted to generate human speech. The theory which is believed behind this is human speech is more complex.

The architecture of MMAudio. It consists of a series of multimodal transformer blocks with visual/text/audio branches, followed by a series of audio-only transformer blocks. For synchrony, they devised a conditional synchronization module that extracts and integrates into the generation process for temporal alignment.
Note: The figure is a conceptual representation and may not reflect the exact architecture used in the paper.

Code and Demonstrations

Feature Extraction Pipeline

# Simplified audio feature extraction
class AudioFeatureExtractor:
    def __init__(self, model_name):
        self.clip_model = load_clip_model()
        self.vae = load_vae_model()
        self.mel_converter = MelConverter()

    def process(self, waveform):
        mel = self.mel_converter(waveform)
        latents = self.vae.encode(mel)
        text_features = self.clip_model.encode_text(latents)
        return latents, text_features

# Simplified video feature extraction
class VideoFeatureExtractor:
    def __init__(self):
        self.synchformer = load_synchronization_model()
        self.clip_encoder = load_video_encoder()
        
    def process(self, video_frames):
        clip_features = self.clip_encoder(video_frames)
        sync_features = self.synchformer(video_frames)
        return clip_features, sync_features
                

Multimodal Fusion Architecture

class MultimodalTransformer(nn.Module):
    def __init__(self):
        super().__init__()
        self.cross_attn = nn.MultiheadAttention(embed_dim=1024, num_heads=16)
        self.temporal_conv = nn.Conv1d(1024, 1024, kernel_size=3)
        self.adaLN = AdaptiveLayerNorm(1024)
        
    def forward(self, video_feats, audio_feats, text_feats):
        # Joint attention across modalities
        combined = self.cross_attn(
            query=audio_feats,
            key=torch.cat([video_feats, text_feats], dim=1),
            value=torch.cat([video_feats, text_feats], dim=1)
        )
        # Temporal processing
        aligned = self.temporal_conv(combined)
        # Conditional normalization
        output = self.adaLN(aligned)
        return output

Training and Evaluation

def train_step(batch, models):
    # Extract multimodal features
    audio_feats = models.audio_extractor(batch['waveform'])
    video_feats = models.video_extractor(batch['video_frames'])
    text_feats = models.text_encoder(batch['text'])
                    
    # Multimodal fusion
    fused_feats = models.fuser(video_feats, audio_feats, text_feats)
                    
    # Flow matching objective
    noise = torch.randn_like(fused_feats)
    t = torch.rand(fused_feats.size(0))
    x_t = t[:,None,None]*fused_feats + (1-t[:,None,None])*noise
    pred_velocity = models.velocity_net(x_t, t)
    loss = F.mse_loss(pred_velocity, fused_feats - noise)
                    
    return loss

Demo

Check out the demo of MMAudio in action. This was done in Colab environment:

# Colab demo for MMAudio
!nvidia-smi
# Check GPU availability
import torch
if torch.cuda.is_available():
    print('GPU is available')
    device = 'cuda'
else:
    print('GPU not available, using CPU')    
    device = 'cpu'
    
# Install dependencies
!pip install torch torchvision torchaudio transformers
!git clone https://github.com/hkchengrex/MMAudio.git
%cd MMAudio
!pip install -e .
    
# Example data:
%cd /content/MMAudio
!curl https://i.imgur.com/8xHJTzI.mp4 -O
from IPython.display import HTML
from base64 import b64encode
data_url = "data:video/mp4;base64," + b64encode(open('video.mp4', 'rb').read()).decode()
HTML('<video width="400" height="300" controls><source src="' + data_url + '" type="video/mp4"></video>')    
    
# Load the model
!python demo.py --duration=10 --video=video.mp4 --prompt "waves and seagulls"
data_url = "data:video/mp4;base64," + b64encode(open('./output/video.mp4', 'rb').read()).decode()
HTML('<video width="400" height="300" controls><source src="' + data_url + '" type="video/mp4"></video>')    
        

Experiments

Main Results

Video-to-audio

Our smallest model (157M) outperforms prior methods on the VGGSound test set (~15K videos) across most metrics while being computationally efficient. Larger models show improved FD_PaSST and IB-scores, though with diminishing returns likely due to data quality limitations. We evaluate 8-second generations following Wang et al.

Table 2. Onset accuracy, average precision (AP), and F1-score on Greatest Hits, with DeSync on VGGSound for reference.

Method	Acc. ↑	AP ↑	F1↑	DeSync↓
Frieren [67]	0.6949	0.7846	0.6550	0.851
V-AURA [65]	0.5852	0.8567	0.6441	0.654
FoleyCrafter [73]	0.4533	0.6939	0.4319	1.225
Seeing&Hearing [69]	0.1156	0.8342	0.1591	1.204
MMAudio-S-16kHz	0.7637	0.9010	0.7928	0.483
MMAudio-S-44.1kHz	0.7150	0.9097	0.7666	0.444
MMAudio-M-44.1kHz	0.7226	0.9054	0.7620	0.443
MMAudio-L-44.1kHz	0.7158	0.9064	0.7535	0.442

Text-to-audio

Without fine-tuning, our multimodal framework demonstrates state-of-the-art semantic alignment (CLAP) and audio quality (IS) on the AudioCaps test set, despite not being primarily designed for this task.

Table 3. Text-to-audio results on the AudioCaps test set. For a fair comparison, we follow the evaluation protocol of [12] and transcribe all baselines directly from [12], who have reproduced those results using officially released checkpoints under the same evaluation protocol.

Method	Params	FDPANNs↓	FDVGG↓	IS↑	CLAP↑
AudioLDM 2-L [39]	712M	32.50	5.11	8.54	0.212
TANGO [1]	866M	26.13	1.87	8.23	0.185
TANGO 2 [43]	866M	19.77	2.74	8.45	0.264
Make-An-Audio [16]	453M	27.93	2.59	7.44	0.207
Make-An-Audio 2 [15]	937M	15.34	1.27	9.58	0.251
GenAU-Large [12]	1.25B	16.51	1.21	11.75	0.285
MMAudio-S-16kHz	157M	14.42	2.98	11.36	0.282
MMAudio-S-44.1kHz	157M	15.26	2.74	11.32	0.331
MMAudio-M-44.1kHz	621M	14.38	4.07	12.02	0.351
MMAudio-L-44.1kHz	1.03B	15.04	4.03	12.08	0.348

Figure 3. We visualize the spectrograms of generated audio (by prior works and our method) and the ground-truth. Note our method generates the audio effects most closely aligned to the ground-truth, while other methods often generate sounds not explained by the visual input and not present in the ground-truth.

Reflections