Top-Down Semantic Vocoder for LLMs

A Dual-Rate Architecture for Context-Aware Language Modeling

A PyTorch implementation bridging Digital Signal Processing (DSP) multirate theory with discrete text generation. This repository introduces a decoupled LLM architecture that separates high-level semantic reasoning (the "Planner") from low-level grammatical synthesis (the "Vocoder").

This approach natively solves the $O(N^2)$ context-window bottleneck of standard autoregressive transformers by treating text generation as an acoustic synthesis problem: upsampling a slow-rate semantic control signal into a fast-rate discrete token sequence.

---

Research Lineage

This repository is the direct evolution of the foundational experiments in eladwf/adaptive-multirate-transformers. While the previous work explored exploratory LFO wrappers and adaptive hyperparameters on character-level models, this repository scales those DSP principles into a highly structured, sentence-level semantic vocoder for BPE tokenization.

---

The Problem: Monolithic Autoregression

Standard dense LLMs (like GPT and LLaMA) treat text generation as a flat, sequence-to-sequence math problem. The model spends the exact same amount of compute predicting the letter "e" at the end of "the" as it does calculating the logical crux of a mathematical proof.

Because self-attention scales quadratically ( $O(N^2)$ ), attempting to hold long-term narrative or logical consistency across 100,000+ tokens becomes computationally crippling.

The Solution: Multirate Semantic Synthesis

In Text-to-Speech (TTS), it is standard practice to decouple the pipeline: a model generates a low-sample-rate continuous signal (a mel-spectrogram), which is then fed into a high-sample-rate Vocoder (like WaveNet) to synthesize discrete audio samples.

This repository applies that exact paradigm to discrete text (BPE/characters) using a dual-rate architecture:

1. The Semantic Planner (Slow Rate): A multirate sentence-level autoregressive semantic planner. It operates on highly compressed semantic embeddings and predicts the next sentence's embedding exactly at the 1/C token rate.
2. The Lexical Vocoder (Fast Rate): An autoregressive GPT that learns high-frequency local grammar and spells out the tokens using an $O(N \cdot W)$ sliding-window local attention mechanism, strictly modulated by the top-down Planner.

---

Block Architecture Diagram

My dual-track mechanism processes text at two independently optimized clock rates. A slow, sequence-level top track provides dense conceptual conditioning, while the fast autoregressive bottom track emits discrete sequence tokens. The semantic vector "vocodes" the generation at key adapter bottlenecks.

 [Top Track: Slow Rate 1/C]                [Bottom Track: Fast Rate]
 
      Sentence Embeddings                         BPE Tokens
          │                                           │
          ▼                                           ▼
[Autoregressive Planner]                        [GPT Blocks (Sliding Window)]
          │                                           │
          ▼                                           ▼
  (Future Semantic Vector) ══════╗             (Hidden States)
                                 ║                    │
                                 ▼                    ▼
                      [ _TopDownAdapterBlock (Conditioning Phase) ]
                                                      │
                                                      ▼
                                            (Next Token Emission)

---

Architecture Highlights

1. The Residual Logit Delta

Rather than concatenating semantic conditions into the prompt or embedding layer—which often collapses the base model's language capabilities—this architecture uses a late-stage adapter fusion.

The base GPT computes the local grammar unimpeded via its fast sliding-window blocks. The TopDownAdapterBlock then calculates a "delta" probability distribution based on the global semantic phase. The final output is a feed-forward correction:

$$\mathbf{Logits}{final} = \mathbf{Logits}{base} + \text{softplus}(\alpha) \cdot \mathbf{Logits}_{delta}$$

2. Time-Varying Semantic Phase Alignment

To prevent the control signal from becoming a static DC offset, the architecture implements dynamic upsampling (upsample_add or cross_attn). The slow-rate semantic timeline is interpolated to match the fast-rate lexical sequence, ensuring the vocoder knows exactly which conceptual "frame" it is currently inside during step-by-step autoregressive generation.

---

Theoretical Context Window Multiplier

By running the heavy reasoning over a compressed semantic sequence, this architecture acts as a massive context-window multiplier.

Let $N$ be the total sequence length in BPE tokens. Let $C$ be the compression ratio (the average number of tokens per semantic state, e.g., $C \approx 15$ for sentence-level embeddings). Let $W$ be the local sliding window size for the vocoder.

Standard GPT Attention Complexity: $$O(N^2 \cdot d)$$

Multirate System Complexity: The attention cost is split between the slow-rate Planner and the fast-rate Vocoder: $$O_{planner} = \left(\frac{N}{C}\right)^2 \cdot d$$ $$O_{vocoder} = (N \cdot W) \cdot d$$

Total Logical Complexity: $$O\left( \left(\frac{N}{C}\right)^2 + N \cdot W \right) \ll O(N^2)$$

Example: For a 100,000-token document at $C=20$, the core logical attention operations drop from 10,000,000,000 to just 25,000,000—a 400x theoretical reduction in global attention compute.

---

Hardware Implementation Note

[!NOTE] Current Implementation: This repository validates the logical complexity using banded boolean masks within standard PyTorch nn.MultiheadAttention. Because standard PyTorch still allocates the full attention matrix prior to masking, wall-clock speedups and VRAM reductions are not yet fully realized in this reference codebase. True sub-quadratic hardware scaling requires swapping the base blocks with a block-sparse sparse attention kernel (e.g., FlashAttention-2 or xFormers BlockDiagonalMask).

---

Experimental Insights

Conditioning Over-Reliance & Exposure Bias

[!WARNING] Conditioning Over-Reliance & Exposure Bias: The _TopDownAdapterBlock transmits the semantic signal so efficiently that the base GPT heavily prioritizes latent-decoding over local syntactic learning. Even with 15% Semantic Dropout applied from step zero, the model achieves an unusually high Top-1 accuracy (~85%), indicating it uses the 384D vector as a hash-key for the sentence. Future work will explore aggressive Classifier-Free Guidance (e.g., 50%+ dropout) or continuous noise injection to force the base model to build a more robust, independent grammar engine and prevent repetitive loops during nucleus sampling.

Generation Samples (Nucleus Decoding, Step 8000)

The following sample demonstrate the multirate vocoder generating greedy continuations based on the top-down semantic vectors. This is a raw, unedited outputs from the validation set evaluated by the Ollama LLM judge.

Example: Successful Semantic Alignment

The vocoder successfully internalizes the narrative direction from the semantic vector and generates a fluent, grammatically correct continuation that aligns with the reference plot.

Reference Concept: "...near the pit. One day, Tom lost his red ball. He was very sad. Tom asked his friend, Sam, to help him search for the ball. They look"

Vocoder Output: "...near the pit. One day, Tom lost his red ball. Tom was very sad. Tom went to the west field and asked his friend, Sam, to help him..."

Judge Score: 4/5 (High Coherence & Fluency)

---

Results & Performance

When evaluated on standard narrative datasets (e.g., TinyStories via BPE tokenization), the TopDownSemanticVocoder outperforms the predictive power of a standard autoregressive GPT baseline of equivalent size, achieving better results from the LLM judge.

Empirical Convergence

Below is the training loss convergence curve, demonstrating stable optimization despite the complex dual-rate adapter configuration.

---

Repository Structure

The core pipeline is stripped down for reproducibility and integration:

• llm_dsp_lab/models/topdown_semantic_vocoder.py: The core PyTorch architecture containing TopDownSemanticVocoder and _TopDownAdapterBlock.
• llm_dsp_lab/models/semantic_planner.py: The multirate sentence-level autoregressive semantic planner.
• scripts/prepare_topdown_annotations.py: The data pipeline script. Uses spaCy and SentenceTransformers to extract sentence-level semantics and BPE-aligned POS tags to build the multirate .pt cache. Includes memory-safe sharding for massive corpora.
• scripts/run_experiment.py: A lightweight, standalone training loop demonstrating how to load the multirate cache and calculate the joint loss (logits_loss + planner_loss).

Getting Started

(1) Annotate your dataset:

python scripts/prepare_topdown_annotations.py --dataset tinystories

(2) Train the Vocoder:

python scripts/run_experiment.py configs/topdown_semantic_vocoder_tinystories_bpe.yaml

License

MIT License. Feel free to integrate the TopDownAdapterBlock into your own multirate workflows.

---

Contact

If you have questions, ideas, or want to collaborate, feel free to reach out:

• Email: eladwf@gmail.com

---