⚛️量子位•Stalecollected in 2h
Noiz AI Open-Sources Fast Audio Generation Model
💡A high-performance, open-source audio model with 0.24s latency, perfect for real-time interactive AI applications.
⚡ 30-Second TL;DR
What Changed
Achieves audio generation in only 4 steps
Why It Matters
This model significantly lowers the barrier for real-time audio synthesis, enabling faster integration of high-quality audio into interactive AI agents.
What To Do Next
Download the model weights from the open-source repository and benchmark its latency against existing solutions like AudioLDM.
Who should care:Developers & AI Engineers
Key Points
- •Achieves audio generation in only 4 steps
- •Inference latency of 0.24 seconds on a single GPU
- •Supports timestamp-based audio control and understanding
🧠 Deep Insight
Web-grounded analysis with 16 cited sources.
🔑 Enhanced Key Takeaways
- •The open-sourced model from Noiz AI, developed in collaboration with HKUST and Tsinghua University, achieves its rapid audio generation with an inference latency of 0.24 seconds on a single GPU, making it highly efficient for real-time applications.
- •Beyond basic text-to-speech, the Noiz AI platform, which may leverage this underlying model, offers advanced capabilities such as voice cloning from short audio samples and multilingual dubbing that preserves original timing and emotional nuances.
- •Noiz AI's technology allows for the generation of voices with significant emotional depth, including customizable tones like happy, angry, or sad, and can automatically detect and apply appropriate emotions based on text context.
- •The platform extends its generative capabilities to include one-prompt audio generation for sound effects, background music, and ambient atmospheres, enabling comprehensive soundscape creation from text descriptions.
- •The collaboration between Noiz AI, HKUST, and Tsinghua University is part of a broader strategic partnership between the academic institutions to foster AI talent and research, including joint forums on AI governance and cross-institutional research funds.
📊 Competitor Analysis▸ Show
While the article highlights an open-source model with 0.24s inference latency, the broader 'Noiz AI' platform offers commercial services. The table below compares Noiz AI's general offerings and the specific open-source model's reported latency against leading commercial and open-source audio generation solutions.
| Feature/Product | Noiz AI (Open-Source Model) | Noiz AI (Commercial Platform) | Inworld (Realtime TTS) | Google Gemini 3.1 Flash TTS | ElevenLabs | Cartesia Sonic 3.5 Turbo | Kokoro (Open-Source) |
|---|---|---|---|---|---|---|---|
| Nature | Open-Source Model | Commercial Platform | Commercial API | Commercial API | Commercial API | Commercial API | Open-Source Model |
| Primary Use | Fast Audio Generation | TTS, Voice Cloning, Dubbing, Sound Design | Realtime TTS, Voice Agents | TTS, Google Cloud Integration | High-Fidelity TTS, Dubbing | Low-Latency TTS | Edge/Low-Cost TTS |
| Latency | 0.24s (inference, single GPU) | 1-3s (generation) / sub-250ms (P90 end-to-end) | sub-130ms (Mini) / sub-250ms (Max) P90 end-to-end | Not specified for inference, competitive end-to-end | Not specified, but streaming-native | ~40ms TTFB (Time To First Byte) | 36x real-time (Colab T4) / 5x (32-core CPU) |
| Key Features | 4 steps generation, timestamp control | Emotional voices, voice cloning, multilingual dubbing, sound design | Natural-language steering, zero-shot voice cloning | High quality, native GCP integration | 70+ languages, voice isolation, sound effects | Ultra-low latency | 82M params, 8 languages, no voice cloning |
| Pricing | Free (open-source) | Paid tiers (e.g., from $4.50/month) | From $25/1M characters | $36.6/1M characters | From $50/1M characters | Not specified | Free (self-hosted) |
🛠️ Technical Deep Dive
While specific architectural details for the open-sourced model achieving 0.24s latency in 4 steps are not fully detailed in the search results, the broader Noiz AI platform emphasizes several technical capabilities:
- Emotional Depth and Nuance: The platform generates audio with natural pauses, breathing sounds, and accurate intonation to mimic human speech patterns, offering adaptable voice outputs that can express specific emotions like happiness, sadness, anger, or excitement.
- Voice Cloning: It allows users to create digital replicas of specific voices, often from short audio samples (e.g., 3-15 seconds), ensuring consistency across content.
- Multilingual Dubbing: The system seamlessly translates video content into multiple languages while preserving the original emotional tone and timing.
- Sound Design: Noiz AI supports one-prompt generation of dialogues, sound effects (SFX), background music (BGM), and ambient atmospheres, suggesting a comprehensive audio AI model capable of handling various sound elements.
- Academic Context: Research from HKUST, a collaborator, includes the development of 'AudioX,' a model that generates high-quality audio and music from diverse inputs (text, video, images) using a diffusion transformer architecture with a multi-modal masking strategy. This approach creates a unified representation space across different data types. This suggests a potential underlying architectural direction for the open-sourced model, although a direct link to the '4 steps' is not explicitly stated.
🔮 Future ImplicationsAI analysis grounded in cited sources
The open-sourcing of a highly efficient audio generation model will significantly lower the barrier to entry for developers.
By providing a fast, accessible model, more developers can integrate advanced audio capabilities into applications without extensive proprietary research or high costs, fostering innovation in areas like real-time AI assistants and interactive media.
The low inference latency will accelerate the development of truly real-time interactive AI applications.
A 0.24-second inference latency on a single GPU is crucial for applications requiring immediate audio feedback, such as live gaming, conversational AI, and instant dubbing, making interactions feel more natural and responsive.
The academic collaboration and open-source nature will drive rapid advancements and community contributions in audio AI research.
The involvement of HKUST and Tsinghua University, combined with an open-source release, encourages broader scientific scrutiny, community development, and the integration of diverse research perspectives, potentially leading to faster improvements and new applications.
⏳ Timeline
2023
HKUST and Tsinghua University co-hosted the 'International AI Cooperation and Governance Forum 2023', fostering broader AI collaboration.
2025-02-18
Noiz Studio, a platform offering AI dubbing, speech cloning, text-to-speech, and video translation, launched on Product Hunt.
2025-05-07
A HKUST team led by Prof. XUE Wei unveiled AudioX, an innovative model for high-quality audio and music generation using a diffusion transformer architecture.
2025-11-12
Noiz AI, described as an all-in-one AI agent for voice, dubbing, and audio magic, launched on Product Hunt.
2026-02-14
Noiz.ai was reviewed as a 'next-generation text-to-speech system' highlighting voice cloning, emotional control, and dubbing capabilities.
📎 Sources (16)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: 量子位 ↗