The Rise of Native Speech Language Models: Beyond Cascaded Audio Pipelines

2026-08-04

The transition from cascaded AI voice pipelines to native end-to-end Speech Language Models (SpeechLMs) marks a structural paradigm shift in human-computer interaction. By leveraging neural audio codecs and novel architectural paradigms, modern AI models can now process continuous audio streams with near-human latency and rich paralinguistic fidelity.

The End of the Cascaded Pipeline: The traditional paradigm of chaining Automatic Speech Recognition (ASR), Large Language Models (LLMs), and Text-to-Speech (TTS) is rapidly being replaced by native, end-to-end Speech Language Models (SpeechLMs) that process audio directly, eliminating compound latency and the catastrophic loss of paralinguistic cues.
The Codec Bottleneck Resolved: Bridging the continuous nature of audio with the discrete token-based architecture of Transformers has been solved by Neural Audio Codecs utilizing Residual Vector Quantization (RVQ), allowing for ultra-low bitrate, high-fidelity acoustic compression.
Architectural Divergence: The field is currently split between interleaved text-audio architectures (like GLM-4-Voice and Spirit-LM) which mix modalities in a single sequence, and parallel multi-stream architectures (like Kyutai's Moshi) which utilize an "Inner Monologue" to synchronize semantic planning with real-time acoustic output.
Factorized Tokenization: Emerging frameworks, notably UniAudio 2.0's ReasoningCodec, are pushing the frontier by explicitly disentangling audio into "reasoning tokens" (for semantic understanding) and "reconstruction tokens" (for acoustic fidelity), solving the historic trade-off between language alignment and audio quality.
The Proprietary Omni-Models: Commercial giants have validated the native multimodal approach with models like GPT-4o and Gemini Live, achieving near-human conversational latencies (sub-300ms) and affective (emotion-aware) dialogue, though their underlying architectures remain closed-source black boxes.

The architecture of human-computer interaction is undergoing a structural paradigm shift. For years, conversational artificial intelligence has been handicapped by a fundamental modality gap: LLMs natively understand discrete text, while the human world operates in continuous, multi-sensory streams. To bridge this divide, engineers constructed fragile, highly latent pipelines that translated voice to text, processed the text, and synthesized the text back to voice. While functional, this approach stripped human communication of its most vital non-verbal elements—tone, cadence, emotion, and overlapping conversational dynamics.

Recent breakthroughs in neural audio compression and multi-modal alignment have catalyzed the rise of Native Audio Language Models. By teaching foundational models to natively ingest, process, and output discrete acoustic tokens, researchers are collapsing the traditional three-stage pipeline into a unified, end-to-end cognitive architecture. This transition is not merely an optimization of latency; it is a fundamental expansion of the LLM's sensory apparatus. It appears highly likely that the next generation of foundational models will treat audio not as a peripheral attachment, but as a primary, native modality equivalent to text.

The Demise of the Cascaded Audio Pipeline

To appreciate the magnitude of native audio language models, one must first deconstruct the limitations of the outgoing standard. Historically, voice-based interactions with artificial intelligence relied on a tripartite cascade: Automatic Speech Recognition (ASR), a Large Language Model (LLM), and Text-to-Speech (TTS) synthesis [cite: 1].

In this legacy framework, an incoming audio wave is transcribed into text by the ASR model. This text is fed as a prompt into the LLM, which generates a textual response. Finally, the TTS engine converts the LLM's text back into an audio waveform. While this pipeline enabled the first wave of voice assistants, it suffers from severe systemic vulnerabilities.

The most immediate flaw is compound latency. Each model in the sequence requires its own compute time, and the LLM cannot begin generating a response until the ASR has finished transcribing the user's audio. This sequential bottleneck creates awkward, turn-taking delays that prevent interactions from feeling like a natural human conversation [cite: 2].

More critically, the ASR-LLM-TTS cascade suffers from severe information loss [cite: 1]. When continuous speech is transcribed into flat text, all paralinguistic information is annihilated. The ASR discards the speaker's tone, emotional state, breathing patterns, background environment, and regional accent. The LLM receives a sterile transcript, entirely blind to whether the user was whispering, shouting, laughing, or crying. Consequently, the TTS system generates a response based solely on semantic text, leading to the notoriously robotic, emotionless delivery characteristic of legacy voice assistants. By directly modeling speech data, native Speech Language Models (SpeechLMs) capture both semantic meaning and paralinguistic information, allowing for richer, contextually and emotionally aware interactions [cite: 1].

The Tokenization Bottleneck: Neural Audio Codecs

If native audio processing is the solution, why was it not implemented sooner? The barrier lies in the fundamental architecture of the Transformer model. Transformers operate on discrete tokens—finite vocabularies of numbers that represent chunks of data. Text is easily tokenized using methods like Byte-Pair Encoding (BPE), which typically produces vocabularies of 30,000 to 100,000 tokens [cite: 3]. Audio, however, is a continuous, high-dimensional waveform.

Sampling raw audio at standard rates (e.g., 24 kHz) produces 24,000 data points per second. Feeding 24,000 continuous values into a standard Transformer for just one second of audio would immediately overwhelm the model's context window and computational capacity. The solution requires compressing the continuous acoustic waveform into a low-framerate sequence of discrete tokens without losing perceptual quality.

This is the domain of Neural Audio Codecs. A codec (compressor-decompressor) reduces file size for efficient storage and transmission [cite: 3]. While traditional lossy codecs (like MP3 or AAC) rely on psychoacoustic models to discard inaudible frequencies, neural audio codecs leverage deep artificial neural networks to learn the optimal compression representations. Modern neural codecs—such as Google's SoundStream and Meta's Encodec—are proficient in encoding audio signals across a broad spectrum of bitrates, superseding traditionally handcrafted pipelines [cite: 4]. For instance, Google's SoundStream can operate dynamically across variable bitrates ranging from 3 kbps to 18 kbps, providing high-fidelity compression that outperforms older standards like Opus at 12 kbps while using a fraction of the bandwidth [cite: 5, 6]. Similarly, Meta's EnCodec pushes this boundary further by handling diverse audio from 24 kHz mono speech up to 48 kHz stereo music, leveraging bitrates stretching from 1.5 kbps up to 24 kbps [cite: 7].

Residual Vector Quantization (RVQ)

The mathematical heart of state-of-the-art neural audio codecs is a technique known as Residual Vector Quantization (RVQ). To understand RVQ, we must first understand standard Vector Quantization (VQ) and why it fails in high-fidelity audio.

Standard Vector Quantization works by mapping an input vector (a chunk of the audio signal) to the closest matching vector in a predefined "codebook." The input is then represented merely by the integer index of that codebook entry [cite: 8, 9]. For example, if we have a codebook of 1,024 vectors, any continuous audio chunk is approximated by choosing one of the 1,024 options.

The problem with standard VQ is the exponential explosion of required codebook sizes. To accurately represent complex, high-fidelity audio, a single quantizer layer would need an exponentially large codebook to map all possible acoustic variations [cite: 4]. Searching through a codebook of millions of vectors in real-time is computationally intractable.

Residual Vector Quantization offers an elegant, cascaded solution. Instead of one massive codebook, RVQ uses a series of smaller, sequential codebooks to provide a progressively finer approximation of the high-dimensional vector [cite: 4].

The RVQ process executes through the following cascade:

First-Order Quantization: The continuous input vector is projected onto the closest vector in the primary codebook (e.g., a codebook of 1,024 entries). The index of this vector is saved [cite: 4, 8].
Residual Calculation: The system calculates the "residual"—the mathematical difference (the error) between the original continuous vector and the quantized vector chosen from the first codebook [cite: 8].
Sequential Refinement: This residual vector is then passed to a second codebook, which quantizes the error. A new residual is calculated from this second step, and passed to a third codebook, and so on [cite: 4, 8].

By repeating this process (often 8 times), RVQ exponentially increases the effective number of represented clusters. With just 4 codebooks of size 10, it acts akin to the decimal system, giving 10,000 separate categories without needing a flat list of 10,000 entries [cite: 9, 10]. The final RVQ representation is simply an array of the indices that were matched in each sequential codebook [cite: 8].

To ground this abstract mathematical concept, consider the following conceptual pseudo-code illustrating how a continuous vector is quantized through an RVQ cascade:

class ResidualVectorQuantizer:
 def __init__(self, num_quantizers=8, codebook_size=1024):
 # Initialize a cascade of codebooks
 self.codebooks = [Codebook(size=codebook_size) for _ in range(num_quantizers)]
 
 def encode(self, continuous_audio_vector):
 quantized_indices = []
 residual = continuous_audio_vector
 
 for codebook in self.codebooks:
 # Find the nearest vector in the current codebook
 nearest_vector, index = codebook.find_nearest(residual)
 
 # Store the discrete index
 quantized_indices.append(index)
 
 # Update the residual (the error left over)
 residual = residual - nearest_vector
 
 # Return the discrete representation (array of indices)
 return quantized_indices

This additive technique approximates high-dimensional vectors by sequentially summing optimized codewords, enabling fine-grained compression and adaptable rate-distortion trade-offs [cite: 10]. Because the later codebooks in the cascade represent increasingly minute acoustic details, RVQ natively supports variable bitrates. If a system requires lower bandwidth, it can simply drop the indices from the final codebooks in the sequence, sacrificing minimal fidelity for significant data savings [cite: 10].

Recent advancements have further refined this architecture. Variable Bitrate RVQ (VRVQ) adapts the number of codebooks used per frame dynamically [cite: 11]. Instead of employing a fixed number of codebooks (which is suboptimal during moments of silence or simple input audio), VRVQ estimates an importance map and dynamically drops unnecessary codebooks, vastly improving the rate-distortion tradeoff and computational efficiency during inference [cite: 11].

Architecting End-to-End SpeechLMs: The Interleaved Paradigm

With neural audio codecs translating continuous sound into discrete tokens, the architectural challenge shifts to integrating these audio tokens with traditional text tokens inside the LLM. One prominent approach is the Interleaved Modeling Paradigm, championed by models like GLM-4-Voice and Spirit-LM [cite: 12, 13].

In the interleaved paradigm, text tokens and audio tokens share the same sequential input space. The language model alternates between predicting text and predicting audio [cite: 13]. This approach leverages the vast semantic reasoning capabilities pre-trained into the LLM via text, and attempts to align those textual representations directly with acoustic representations.

However, this alignment requires massive amounts of paired data. A model cannot simply deduce how a text token maps to an audio token without observing millions of examples. While Spirit-LM utilized speech-text interleaving data curated from parallel corpora, the sheer scarcity of high-quality, naturally occurring parallel speech-text datasets heavily restricts the scale of this alignment [cite: 12].

To overcome this data wall, researchers behind GLM-4-Voice and Spirit-LM turned to large-scale synthetic data generation [cite: 12, 14].

The synthetic interleaved data pipeline (often referred to as an InSerter approach) typically involves the following stages:

Corpus Extraction: Large-scale text from existing pre-training corpora is collected and cleaned [cite: 12, 14].
Sentence-Level Sampling: Using punctuation marks as delimiters, specific sentences within the text are randomly sampled to be converted into speech segments [cite: 14].
Synthetic TTS Conversion: The sampled text is converted into audio using high-fidelity TTS models (such as CosyVoice 2.0). To ensure acoustic diversity, tens of thousands of unique multi-speaker voice prompts are applied [cite: 14].
Tokenization and Re-insertion: The synthetic audio is tokenized by the neural codec, and the resulting discrete audio tokens are interleaved back into the original text sequence, replacing the sampled sentences [cite: 14].

By pre-training on 1 trillion tokens—comprising this synthetic interleaved data alongside 700,000 hours of unsupervised, real-world speech data and traditional supervised datasets (ASR and TTS)—models like GLM-4-Voice bridge the modality gap from the input side [cite: 1, 12]. GLM-4-Voice operates using a supervised speech tokenizer with an ultra-low bitrate of 175bps at a 12.5Hz frame rate, utilizing a single-codebook approach [cite: 12]. This single-codebook design avoids the complex architectural adjustments often required for multi-layer speech token generation, effectively preserving the model's text processing capabilities while enabling efficient speech modeling [cite: 12].

Empirical evidence suggests that interleave-paradigm models generally better preserve their core language and reasoning capabilities compared to parallel-paradigm models (which average text and audio representations as a combined input) [cite: 13]. However, the interleaved approach is not without flaws. Because the model must sequentially predict audio tokens alongside text tokens, latency can become an issue, particularly when the audio token rate is high [cite: 13].

Factorizing Audio: UniAudio 2.0 and the ReasoningCodec

A central tension in designing an Audio Language Model is that semantic understanding and acoustic reconstruction demand entirely different types of information. Text tokens (like BPE) contain highly concentrated semantic meaning but zero acoustic data. Conversely, standard RVQ audio tokens contain rich acoustic data but dilute semantic meaning across thousands of frames.

If an LLM is forced to use standard acoustic tokens for both understanding a question and generating the sound of the answer, it struggles. The model wastes computational power modeling background noise, room reverb, and microphone artifacts just to grasp the meaning of a spoken word. This trade-off between language alignment and audio quality led to the development of explicitly factorized tokenizers, most notably the ReasoningCodec used in UniAudio 2.0 [cite: 15, 16].

UniAudio 2.0 treats audio understanding and audio generation as fundamentally distinct but related tasks [cite: 15]. The model introduces a discrete audio tokenizer that factorizes the audio waveform into two separate streams of tokens [cite: 16, 17]:

Reasoning Tokens: These tokens encode text-aligned, high-level analysis and planning representations. They strip away acoustic noise to focus entirely on the semantic meaning of the audio, allowing the LLM to understand spoken language just as easily as written text [cite: 15, 16].
Reconstruction Tokens: These tokens encode the semantic-rich acoustic cues required for high-fidelity waveform reconstruction. They handle the paralinguistics, timbre, and environmental sounds [cite: 15, 16].

This design achieves understanding performance comparable to strong continuous representation encoders (like WavLM or Whisper encoders, which preserve rich perceptual information but cannot generate audio) while simultaneously improving generation quality over prior discrete tokenizers [cite: 15, 16].

Architecturally, UniAudio 2.0 abandons the naive unified autoregressive transformer approach (where all layers indiscriminately process both text and audio). Instead, it implements a functionally specialized autoregressive architecture over its 100 Billion text tokens and 60 Billion audio tokens of training data [cite: 16, 18]. The network is divided vertically: lower layers are specialized as audio understanding experts, middle layers act as cross-modal alignment experts (initialized from a pretrained LLM), and upper transformer layers serve as audio generation experts [cite: 16]. This functional layer specialization allows UniAudio 2.0 to perform competitively on in-domain evaluations while demonstrating strong few-shot and zero-shot generalization to unseen tasks across speech, sound, and music [cite: 15, 16, 19].

Full-Duplex Conversational Dynamics: Deconstructing Moshi

While interleaved and factorized architectures solve the problem of audio representation, they do not inherently solve the problem of conversational dynamics. Human conversation is not a walkie-talkie exchange of turns. It is fluid, overlapping, and "full-duplex"—meaning humans listen and speak simultaneously, utilizing backchannels ("mm-hmm," "yeah") and interruptions seamlessly.

Achieving this in AI requires moving beyond standard turn-based generation. Enter Moshi, a 7-Billion parameter real-time, full-duplex speech-to-speech foundation model developed by Kyutai [cite: 3, 20]. Moshi operates with a theoretical latency of 160ms and a practical overall latency as low as 200ms on an L4 GPU—an ultra-low-power, datacenter-grade enterprise GPU featuring 24GB of GDDR6 VRAM and a mere 72-watt thermal design power (TDP), heavily optimized for cost-efficient AI inference [cite: 20, 21, 22, 23]. This capability completely eliminates the pipelined delays of legacy ASR/TTS systems.

Moshi achieves this through a multi-stream architecture that models audio from the user and audio from the AI on separate, parallel channels [cite: 24]. The system is built on three core components:

1. The Mimi Neural Audio Codec

Mimi is a state-of-the-art neural audio codec developed specifically to interface with LLMs [cite: 3, 24]. Improving upon SoundStream and Encodec, Mimi jointly models semantic and acoustic information using distillation [cite: 24]. It processes 24 kHz mono audio down to a 12.5 Hz representation, resulting in a remarkably low bandwidth of 1.1 kbps [cite: 20, 21].

Crucially, Mimi is fully causal (streaming) with an 80ms frame size, making it ideal for a streaming Transformer. It maintains a compact audio vocabulary of 2,048 tokens per quantizer [cite: 3, 21, 24]. Its performance exceeds existing non-streaming codecs like SpeechTokenizer and SemantiCodec through improved architecture and adversarial training (akin to a Generative Adversarial Network) [cite: 21, 24, 25].

2. Multi-Stream Audio Modeling

Unlike sequential interleaved models, Moshi stacks the tokens of the AI and the user for each timestep [cite: 24]. This means the model continuously predicts its own acoustic output while simultaneously ingesting the user's acoustic input. This enables the modeling of full-duplex conversational dynamics, accurately handling overlaps, backchanneling, and sudden interruptions [cite: 24]. If a user interrupts Moshi mid-sentence, the model natively processes the incoming user tokens on the parallel stream and can dynamically halt or alter its own output tokens without requiring an external stop-generation command.

3. The Inner Monologue

Generating raw acoustic tokens directly from conversational context is highly prone to hallucination and logical collapse, as the model must dedicate immense compute to acoustic rendering rather than reasoning. Moshi solves this by introducing a parallel text stream known as the "Inner Monologue" [cite: 20, 24, 25].

Alongside the two audio streams, Moshi predicts text tokens corresponding to its own upcoming speech [cite: 21]. This text stream acts as a cognitive scratchpad, allowing the model to formulate a semantic plan just moments before synthesizing it into Mimi audio tokens. Ablation studies demonstrate that this Inner Monologue massively improves the quality of the model's generation, making the AI smarter while remaining fully compatible with low-latency streaming [cite: 25]. Furthermore, manipulating these parallel streams yields fascinating byproducts: delaying the audio tokens relative to the text creates a streaming TTS system, while delaying text tokens yields a streaming ASR with precise temporal alignment [cite: 24].

Hardware Constraints and Production Deployment

While native audio models eliminate the compounding latency of cascaded pipelines, their end-to-end architectures introduce new hardware footprint considerations for local and edge deployment. Operating a 7-Billion parameter foundation model like Moshi natively requires significant localized compute. For real-time inference using unquantized FP16 precision, Moshi demands 16 GB to 20 GB of VRAM per active session [cite: 20, 26]. However, by applying weight quantization (such as 8-bit or 4-bit INT quantization), developers can compress this footprint to approximately 8 GB to 10 GB of VRAM, making it viable to run on consumer-grade hardware or dense cloud nodes [cite: 20, 26]. When deployed in production environments on standard datacenter hardware like the NVIDIA L4 (24GB VRAM), a single GPU can comfortably support one real-time, full-duplex session with sufficient headroom for KV cache overhead [cite: 22, 26, 27]. For higher concurrency, enterprise deployments typically scale out horizontally or transition to heavier architectures like the L40S or H100 SXM5 [cite: 26].

The Proprietary Frontier: GPT-4o and Gemini Live

While open-source architectures like Moshi, UniAudio 2.0, and GLM-4-Voice map the academic frontier, the commercial deployment of native audio models is currently dominated by proprietary "Omni" models, specifically OpenAI's GPT-4o and Google's Gemini Live. Precise architectural blueprints, exact parameter counts, and codec specifics for these models remain closed black-boxes. However, based on their API behaviors, latency metrics, and technical disclosures, we can analyze how they implement the native multimodal paradigm.

OpenAI's GPT-4o: The Multimodal Polyglot

Announced as a flagship step toward natural human-computer interaction, GPT-4o ("o" for omni) abandons the legacy pipeline entirely. It natively accepts any combination of text, audio, image, and video as input, and generates combinations of text, audio, and image as output in real-time [cite: 28, 29].

The elimination of the ASR/TTS cascade is evident in its latency metrics. GPT-4o responds to audio inputs in as little as 232 milliseconds, with an average response time of 320 milliseconds—virtually indistinguishable from natural human conversational response times [cite: 28, 29]. Because the model processes raw audio natively, it possesses profound "affective dialogue" capabilities. It detects user emotion, interprets tone, and generates expressive, non-robotic speech with varying intonations, making interactions feel highly empathetic [cite: 28]. In comparative evaluations, GPT-4o operates as a versatile "jack-of-all-trades," excelling in the raw dexterity of cross-modal reasoning (e.g., observing a live video feed while simultaneously conversing audibly about the visual scene) [cite: 28, 30].

Google's Gemini Live: Unified Native Audio Architecture

Google's response, Gemini Live (powered by the Gemini 3.1 Flash Native Audio model), similarly represents a fundamental architectural shift away from rigid, multi-stage voice systems [cite: 2, 31]. Accessed via a single WebSocket connection in Vertex AI or Google AI Studio, Gemini Live processes raw audio natively through a unified, low-latency model [cite: 2, 32].

Gemini Live heavily emphasizes real-time multimodality integrated deeply into the user's broader ecosystem, allowing the agent to converse about topics informed by live streams of visual data (like charts or device screens) simultaneously with spoken input [cite: 2, 30, 33]. Crucially, under optimal network conditions, the Gemini 3.1 Flash Live API consistently targets a sub-400ms response latency, achieving a first-audio latency ranging from 250ms to 500ms, with a median response time of approximately 320ms [cite: 31, 34]. In specialized tasks such as continuous speech-to-speech translation, the latency is intentionally buffered to a constant lag of roughly 3 seconds (median 2,947 ms) to gather enough semantic context before emitting translated audio [cite: 32]. Because the model processes raw audio natively and begins streaming the output chunks before the full response is generated, Gemini Live's interactions often feel even faster than the raw quantitative metrics suggest [cite: 31].

Comparative Architectural Summary

The rapid divergence in native audio architectures has produced distinct paradigms across open-source and proprietary boundaries. The following matrix summarizes the core structural differences between the leading foundational SpeechLMs defining the current frontier:

Model Name	Architecture Paradigm	Primary Codec / Audio Encoding	End-to-End Latency Profile	Open / Proprietary Status	Native Modality Scope
GLM-4-Voice	Interleaved Sequence (Text & Audio)	Single-codebook tokenizer (175 bps)	Variable (Sequential turnaround delays)	Open-Weights (Apache 2.0)	Text / Audio
UniAudio 2.0	Factorized Tokenization (Specialized layers)	ReasoningCodec (Semantic & Acoustic)	Variable (Autoregressive generation)	Open-Source	Text / Audio / Music
Moshi (7B)	Parallel Multi-stream (Full-Duplex)	Mimi Codec (1.1 kbps, 12.5 Hz)	~160ms theoretical / ~200ms practical	Open-Weights (Apache 2.0)	Text / Audio
OpenAI GPT-4o	Unified Omni-Model	Proprietary Black Box	232ms minimum / ~320ms average	Proprietary Closed-Source	Text / Audio / Image / Video
Gemini 3.1 Flash Live	Unified Native Audio (Multimodal)	Proprietary Black Box	250ms - 500ms (Median ~320ms)	Proprietary Closed-Source	Text / Audio / Video

Synthesis and the Path Forward

The architectural transition from cascaded pipelines to native audio language models is not a mere optimization; it is a redefinition of how artificial intelligence perceives and interacts with the physical world. By leveraging Residual Vector Quantization and Neural Audio Codecs, continuous human speech is effectively translated into the discrete mathematical language of Transformers without sacrificing the rich, paralinguistic tapestry of human emotion.

The divergent approaches emerging today—the interleaved semantic alignment of GLM-4-Voice [cite: 12], the explicit factorized reasoning of UniAudio 2.0 [cite: 15], and the multi-stream, full-duplex dynamics of Moshi [cite: 24]—highlight a field undergoing explosive, simultaneous innovation. These open-source endeavors serve to validate the otherwise opaque mechanics powering commercial titans like GPT-4o and Gemini Live.

As these native audio architectures mature, the implications for Human-Computer Interaction are profound. We are moving beyond the era of the "voice assistant" acting as a clunky, turn-based search engine. The immediate future promises ubiquitous, real-time, full-duplex cognitive agents capable of empathetic listening, instantaneous interruption handling, and deep cross-modal reasoning. The modality gap is closing, and the machines are finally learning to listen in high fidelity.

Sources & References