Tencent’s Hunyuan Video-Foley Brings Realistic Sound to AI-Generated Video

For all the advances in AI-generated video, one thing has often been missing: sound that feels real. Tencent’s Hunyuan lab believes it has found the answer with Hunyuan Video-Foley, a new model designed to generate lifelike audio that syncs seamlessly with on-screen action.

HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation

Recent advances in video generation produce visually realistic content, yet the absence of synchronized audio severely compromises immersion. To address key challenges in video-to-audio generation, including multimodal data scarcity, modality imbalance and limited audio quality in existing methods, we propose HunyuanVideo-Foley, an end-to-end text-video-to-audio framework that synthesizes high-fidelity audio precisely aligned with visual dynamics and semantic context. Our approach incorporates three core innovations: (1) a scalable data pipeline curating 100k-hour multimodal datasets through automated annotation; (2) a representation alignment strategy using self-supervised audio features to guide latent diffusion training, efficiently improving audio quality and generation stability; (3) a novel multimodal diffusion transformer resolving modal competition, containing dual-stream audio-video fusion through joint attention, and textual semantic injection via cross-attention. Comprehensive evaluations demonstrate that HunyuanVideo-Foley achieves new state-of-the-art performance across audio fidelity, visual-semantic alignment, temporal alignment and distribution matching. The demo page is available at: https://szczesnys.github.io/hunyuanvideo-foley/.

arXiv.orgSizhe Shan

Why Sound Matters in AI Video

Visuals alone rarely tell the full story. In filmmaking, the rustle of clothing, the crack of thunder, or the clink of a glass are brought to life by Foley artists, experts who recreate everyday sounds with precision. But until now, AI systems have struggled to replicate this craft.

The problem, researchers say, is “modality imbalance.” Many video-to-audio models focused too heavily on text prompts while ignoring the details of the video itself. For example, a clip of a busy beach described only as “waves” might generate ocean sounds—but miss the footsteps in the sand or the cry of seagulls. The result felt flat and artificial.

Tencent’s Three-Part Solution

Tencent’s team tackled the challenge on several fronts:

1. Building a better training set: They compiled a 100,000-hour dataset of video, audio, and text, carefully filtering out poor-quality clips with muffled or missing sound.
2. Smarter AI architecture: The system first locks onto the exact timing of visual cues—like matching a footstep to the moment it hits the pavement—before layering in context from text prompts. This ensures both accuracy and atmosphere.
3. High-quality sound alignment: Using a strategy called Representation Alignment (REPA), the model constantly compares its output against professional-grade audio features, guiding it toward cleaner, richer, and more stable results.

Tested Against the Best

When evaluated against other leading models, Hunyuan Video-Foley consistently outperformed competitors. Human listeners rated its audio as more natural, better timed, and more in tune with the visuals. Technical benchmarks confirmed the improvements.

What This Means for Creators

The breakthrough could have far-reaching implications for content creators. From independent animators to professional filmmakers, anyone working with AI-generated video may soon have access to audio that doesn’t just fill silence but deepens immersion.

By bringing the artistry of Foley into the realm of automation, Tencent’s model helps close the gap between today’s experimental AI clips and tomorrow’s fully immersive media.