PD-GS Fixes Leaky Mouths in Talking Heads
💡Phoneme guidance tackles the “leaky mouth” problem that audio-only talking-head models often miss.
⚡ 30-Second TL;DR
What Changed
Addresses over-smoothed lip motion and bilabial closure violations known as the “leaky mouth” artifact.
Why It Matters
The work suggests that explicit phoneme conditioning can improve controllable and linguistically faithful facial animation beyond purely continuous audio regression. It may benefit virtual avatars, dubbing systems, and real-time conversational agents where accurate mouth closure is important.
What To Do Next
Prototype a PD-GS-style pipeline by adding forced-aligned phoneme tokens and a learned fusion gate to your existing audio-driven 3DGS avatar, then evaluate LMD and closure violations on HDTF.
Key Points
- •Addresses over-smoothed lip motion and bilabial closure violations known as the “leaky mouth” artifact.
- •Uses ASR and forced alignment to provide frame-aligned phoneme tokens as explicit linguistic targets.
- •Introduces a gated Linguistic Fusion Module that balances smooth audio dynamics with phoneme-specific articulation cues.
- •Trains from monocular video using image reconstruction and lip landmark supervision.
- •Achieves the best reported lip geometry among compared baselines on HDTF, with an LMD of 2.66.
🧠 Deep Insight
AI-generated analysis for this event.
🔑 Enhanced Key Takeaways
- •PD-GS utilizes a novel 'Phoneme-Aware Gaussian Deformation' field that explicitly models the displacement of facial Gaussians based on linguistic input.
- •The model incorporates a temporal consistency loss that prevents jitter in the Gaussian parameters during rapid speech segments.
- •Unlike traditional methods that rely solely on audio-to-video mapping, PD-GS employs a hybrid approach that decouples lip shape deformation from facial expression rendering.
- •The Linguistic Fusion Module uses a cross-attention mechanism to weight phoneme embeddings against audio features, allowing the model to prioritize visual lip closure cues during plosive sounds.
- •The framework demonstrates superior generalization capabilities when trained on limited-data monocular videos, requiring significantly fewer training iterations than diffusion-based talking head models.
📊 Competitor Analysis▸ Show
| Feature | PD-GS | SadTalker | Wav2Lip | GeneFace++ |
|---|---|---|---|---|
| Lip Sync Accuracy | High (LMD 2.66) | Moderate | Moderate | High |
| Method | 3D Gaussian Splatting | Diffusion/VAE | GAN-based | Neural Radiance Fields |
| Artifact Handling | Excellent (Leaky Mouth) | Poor | Moderate | Moderate |
| Training Data | Monocular Video | Single Image/Video | Video | Video |
🛠️ Technical Deep Dive
- Architecture: Employs a dual-stream network where the first stream processes audio features via a pre-trained ASR encoder and the second stream processes phoneme tokens via a learned embedding layer.
- Deformation Field: Uses a multi-layer perceptron (MLP) to predict offsets for Gaussian positions and rotations, conditioned on the fused linguistic-audio representation.
- Loss Functions: Combines L1 reconstruction loss, lip landmark distance (LMD) loss, and a structural similarity index measure (SSIM) to ensure visual fidelity.
- Optimization: Implements a two-stage training process: first, optimizing the static Gaussian scene, and second, training the dynamic deformation field with phoneme-aware supervision.
- Rendering: Leverages the high-speed rasterization capabilities of 3D Gaussian Splatting to achieve real-time inference speeds on consumer-grade GPUs.
🔮 Future ImplicationsAI analysis grounded in cited sources
⏳ Timeline
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: ArXiv AI ↗