---

title: Universal Attention Engine — Declarative Sparse Attention for Production date: 2026-05-05 status: planning

Universal Attention Engine

Declarative sparse attention for production LLM inference: define the pattern once, compile it into executable attention paths, and carry the same spec through prefill, decode, KV cache policy, quantization, and serving integration.

Results Snapshot

Reported H100 benchmark highlights:

Full evidence pack: github_showcase/

Architecture

Benchmark Graphs

Backend Speedup

Decode Latency And Memory Savings

Dynamic Routing

Quality Versus Sparsity

Prefill Speedup

The Gap

No system currently does: Declarative pattern spec → Compiled kernel → Deployed in serving → KV cache co-design

The Product

A system where you write:

from attn_engine import AttentionSpec, compile_and_deploy

spec = AttentionSpec(
    # Pattern definition
    pattern="block_sparse",
    sink_tokens=32,
    local_window=512,
    
    # KV cache policy
    cache_budget=1024,
    eviction="heavy_hitter",  # or "attention_score", "recency", custom
    
    # Phase-aware
    prefill_kernel="flash",        # use FlashAttention for prefill
    decode_kernel="sparse_gather", # custom sparse kernel for decode
    
    # Per-head config (optional)
    head_policy="auto",  # analyze model to assign per-head budgets
)

# Compile to efficient kernels for both phases
engine = spec.compile(model_config="llama-70b", hardware="h100")

# Deploy into serving stack
engine.register_backend("vllm")  # or "sglang", "tgi"

Architecture

┌─────────────────────────────────────────────────────┐
│  Layer 1: Declarative Spec (Python DSL)             │
│  - Pattern types (sparse, sliding, coarse-to-fine)  │
│  - KV cache policy (budget, eviction, compression)  │
│  - Per-head/per-layer configuration                 │
│  - Phase annotations (prefill vs decode)            │
└─────────────────┬───────────────────────────────────┘
                  │
┌─────────────────▼───────────────────────────────────┐
│  Layer 2: Compiler (generates phase-aware kernels)  │
│  - Prefill: block-sparse Triton/FlexAttention       │
│  - Decode: gather-based sparse KV kernel            │
│  - Validates correctness (reference impl)           │
│  - Hardware-specific optimization (H100/A100/MI300) │
└─────────────────┬───────────────────────────────────┘
                  │
┌─────────────────▼───────────────────────────────────┐
│  Layer 3: Runtime (integrates with serving stacks)  │
│  - vLLM backend plugin                             │
│  - SGLang backend plugin                           │
│  - KV cache lifecycle management                   │
│  - Continuous batching compatibility               │
│  - PagedAttention-compatible memory layout         │
└─────────────────────────────────────────────────────┘

Key Design Decisions

1. Don't reinvent kernels — compose existing ones

• Prefill: leverage FlexAttention/FlashAttention (already optimal for dense)
• Decode: generate gather-based Triton kernels (sparse KV read)
• Block-sparse prefill: use existing block-sparse flash variants

2. KV cache is part of the spec, not separate

• Pattern determines what to cache
• Eviction policy is declared alongside attention pattern
• Cache format (paged, block-sparse, quantized) is co-optimized

3. Phase-aware compilation

• Prefill is compute-bound → optimize for FLOPs reduction
• Decode is memory-bound → optimize for bytes loaded
• Same logical pattern, different physical kernels

4. Serving-first design

• Must work with continuous batching (different requests, different patterns)
• Must work with PagedAttention memory management
• Must support dynamic batch composition

Content-Dependent Dynamic Routing (Novel)

Existing frameworks assume patterns are known at compile time (positional masks). We support runtime token-content-dependent routing:

spec = AttentionSpec(
    pattern="dynamic_routing",
    router=lambda q, k_blocks: top_k_blocks(q @ mean(k_blocks), k=16),
    # Router runs at block granularity (cheap)
    # Selected blocks get full fine-grained attention
)

This is the coarse-to-fine approach (like DeepSeek NSA) but:

• Training-free (works on any pretrained model)
• Declarative (user specifies routing logic, compiler handles the rest)
• Integrated with KV cache (only cache/load selected blocks)

Competitive Landscape

• FlexAttention: Our pattern spec could compile DOWN to FlexAttention for supported patterns
• FlashInfer: Our runtime layer could USE FlashInfer as a backend
• vLLM/SGLang: We integrate AS a backend, not replace them

We're the glue layer — not competing with any single tool, bridging them.

Milestones

Phase 1: Proof of concept (2-4 weeks)

• Implement spec DSL for 3 patterns (dense, sliding_window, sink+local+sparse)
• Compile to Triton kernels (prefill + decode)
• Validate correctness against eager reference
• Benchmark: show compiled kernel matches hand-tuned perf

Phase 2: Serving integration (4-8 weeks)

• vLLM AttentionBackend plugin
• KV cache eviction integrated with pattern spec
• Continuous batching support
• Benchmark: end-to-end serving throughput vs vanilla vLLM

Phase 3: Advanced patterns (8-12 weeks)

• Content-dependent dynamic routing
• Per-head adaptive budgets (auto-analyzed from model)
• Quantized KV cache support
• Multi-GPU communication-aware patterns

Phase 4: Community + adoption

• Open-source release
• Documentation + examples for common patterns
• Integration PRs to vLLM/SGLang
• Benchmark suite showing energy savings

Prior Art to Build On

Why Now

• FlexAttention just shipped FA4 backend (Dec 2024 → production 2025)
• FlashInfer won MLSys 2025 Best Paper, integrated into SGLang/vLLM
• vLLM has pluggable AttentionBackend API
• The ecosystem pieces exist — nobody has assembled them
• Energy costs of LLM serving are becoming a real industry concern