Awesome LLM Token Optimization

A curated list of strategies, tools, papers, and resources for reducing LLM token costs and improving efficiency in production.

Building with LLMs is expensive. An agent processing 10 reasoning steps can consume 50K-100K tokens per task. This list collects everything you need to cut costs by 80-99% without sacrificing quality.

Contents

• Quick Wins
• Prompt Caching
• Batch APIs
• Model Routing
• Prompt Compression
• Context Window Management
• KV Cache Optimization
• Browser Tool Efficiency
• Cost Tracking Tools
• Pricing Comparison
• Prompt Engineering for Efficiency
• Comprehensive Guides
• Academic Papers
• Community Resources

---

Quick Wins

The highest-impact strategies ranked by effort-to-savings ratio:

Combined pipeline: Cache prefix (90%) + route to cheapest model (60-95%) + batch non-urgent (50%) + compress prompts (5-20x) + cache responses (100% on repeats) = 95-99% cost reduction vs. naive approach.

Prompt Caching

Reuse previously-processed prompt prefixes to avoid re-computing the same tokens.

Provider Docs

• Anthropic Prompt Caching - 90% discount, 5min/1hr TTL. Minimum cacheable prefix: 4,096 tokens on Opus 4.6/Haiku 4.5, 1,024 on Sonnet 4.6/Opus 4.8/Sonnet 5.
• Anthropic Caching Announcement - Blog post explaining economics.
• Anthropic Token-Saving Updates - Cache-aware rate limits, simplified caching.
• Anthropic Extended Thinking + Caching - Thinking blocks get cached in tool-use loops.
• OpenAI Prompt Caching - 50% discount, automatic for 1024+ token prompts. Extended cache retention now defaults to 24h on the GPT-5 series (mandatory on GPT-5.5+), keeping prefixes warm far longer.
• OpenAI Prompt Caching Cookbook - Advanced techniques with code.
• Google Gemini Context Caching - Implicit (auto) and explicit caching, 90% discount.
• Google Vertex AI Caching - Enterprise context caching.
• DeepSeek KV Cache - Disk-based, 64-token granularity. V4 Flash cache hits: $0.0028/M vs $0.14/M base (98% savings). Migration: deepseek-chat and deepseek-reasoner aliases retire July 24, 2026 — update to deepseek-v4-flash or deepseek-v4-pro.
• DeepSeek Context Caching on Disk - Announcement of disk-based context caching cutting input cost ~10x on cache hits.

Strategy: Cached Prefix Pattern

Structure prompts so the system prompt + user profile is the first ~2,000 tokens. All subsequent calls share this prefix. For bulk operations (e.g., scoring 50 items): 1x full + 49x at 10% = 88% total savings.

Tools

• autocache - Transparent Anthropic proxy that auto-injects cache_control breakpoints at optimal positions; up to 90% cost and 85% latency reduction.

Batch APIs

50% discounts for non-time-critical requests. Combine with caching for 95% savings.

• Anthropic Message Batches - Up to 10,000 requests, 24hr turnaround.
• Anthropic Batches Announcement - Use cases and GA details.
• OpenAI Batch API - 50% discount, 50K requests per file.
• OpenAI Batch API FAQ - Limits and behavior.
• Google Gemini Batch API - 50% discount, combinable with context caching.
• Google Vertex Batch Prediction - Enterprise batch.
• Curator - Batch-inference and synthetic-data library; batch=True gives ~50% savings across OpenAI/Anthropic/Gemini/Mistral with built-in caching and retries.

Model Routing

Route simple tasks to cheaper models. 80% of typical LLM calls don't need the most expensive model.

Frameworks

• RouteLLM - Open-source LLM router by LMSYS. Trains routers from preference data; 2x+ cost reduction. Note: last commit Aug 2024; LMSYS team shifted focus to Chatbot Arena.
• LiteLLM - SDK + proxy for 100+ LLMs with routing, cost tracking. Strategies: least-busy, cost-based, latency-based.
• NotDiamond - Per-query best-model selection. Note: the Python SDK was archived Dec 2025 (read-only); the notdiamond.ai service remains active.
• Bifrost - 50x faster than LiteLLM; adaptive load balancer, 1000+ models.
• vLLM Semantic Router - System-level signal-driven router for Mixture-of-Models across cloud, data center, and edge. v0.3 "Themis" (June 2026); SAAR adds session-aware model selection cutting model switches 79% in multi-agent deployments.
• LLMRouter - Open-source routing library with 16+ routers (single-round, multi-turn, agentic, personalized) and a unified CLI.
• Portkey AI Gateway - Open-source AI gateway routing to 1,600+ LLMs with guardrails, caching, and load balancing. Acquired by Palo Alto Networks (May 2026); gateway remains open-source under Apache 2.0.
• OpenRouter - Unified API for 300+ models with auto-router.
• Martian Router - Patent-pending; cuts costs 20-97% via "Model Mapping".

Curated Lists

• Awesome AI Model Routing - Comprehensive list of routing approaches.

Research

• RouteLLM paper - LMSYS blog on cost-quality tradeoffs.
• IBM LLM Routers - IBM's research on training routers.
• vLLM Session-Aware Agentic Routing (SAAR) - Router-owned session memory with hard locks around tool loops cuts model switches 79% and estimated cost 78.7% in multi-agent deployments.
• LLM Routing Explained - Intuitive guide.
• vLLM Semantic Router paper - Signal-driven decision routing for Mixture-of-Modality models; composable signal orchestration across heuristic and neural classifiers.

Prompt Compression

Reduce prompt size while preserving information quality.

Tools

• LLMLingua - Up to 20x compression. Coarse-to-fine iterative method. Integrates with LangChain/LlamaIndex.
• Headroom - Compress tool outputs, logs, files, and RAG chunks before they reach the LLM (60-95% fewer tokens); library, proxy, and MCP server. Claude Code/Cursor/Aider compatible.
• code2prompt - Codebase to LLM prompt with token counting.
• RTK - Single-binary Rust CLI proxy that compresses dev-command output 60-90% before it reaches a coding agent's context. Works with Claude Code, Cursor, Copilot, Gemini CLI.
• Tokenade - CLI proxy that compacts command output, MCP tool results, and file reads before a coding agent sends them; 38.9% lower end-to-end session cost above 200k tokens on THOL, a public benchmark run by Tokenade's own author. Free tier, closed source.
• TOON - Token-Oriented Object Notation: a compact, schema-aware encoding for passing JSON-like data to LLMs; 30-60% fewer tokens than JSON on uniform arrays of objects.
• llmtrim - Quality-gated local proxy and MCP server that compresses prompts, tool outputs, and replies before they reach the LLM, reverting any step that doesn't save tokens (project-reported -31% input / -74% output across 112 A/B cases). Rust CLI plus multi-language library bindings and a WebAssembly/JS package.
• lean-ctx - Rust binary context intelligence layer for AI coding agents; 60-90% fewer tokens via shell-output compression and 10 cached-read modes; MCP server with 76 tools and cross-session memory. Works with Claude Code, Cursor, Copilot, Windsurf, Gemini CLI, and 30+ others.
• snip - Single-binary Go CLI proxy that filters shell output before it reaches a coding agent's context (60-90% fewer tokens), with 132 filters defined as declarative YAML data instead of compiled code; 13 agent integrations, via a native shell hook for Claude Code, Cursor, Copilot, Codex, Pi and Grok, and via injected rules files for Gemini CLI, Windsurf, Cline, Kilo Code and Antigravity.

Research

• CompactPrompt - Unified prompt + data compression pipeline.
• Efficient Prompting Survey - Survey of efficient prompting methods.

The full prompt-compression paper table lives in the Academic Papers section below.

Guides

• LLMLingua Research Blog - Microsoft Research deep dive.
• Prompt Compression Tutorial (FreeCodeCamp) - Practical guide with code.
• Prompt Compression Overview (MLM) - 6x to 480x compression ratios.
• Awesome LLM Compression - Curated paper list.

Lossless Compression Principles

Rule-based lossless distillation achieves 3-4:1 compression without any model:

Strip prose transitions, hedging, rhetoric, and common knowledge. Preserve numbers, entities, decisions, constraints, and risks. Transform prose into dense bullets and verbose text into semicolon-joined clauses. Split into 3,000-5,000 token self-contained sections that load independently.

Context Window Management

Key Research

• Context Rot toolkit (Chroma) - Toolkit reproducing how LLMs degrade well before context limits (18 models tested).
• RAG vs Long Context (Elastic) - RAG is 1250x cheaper for many queries.
• Long Context RAG (Databricks) - Degradation after 32K-64K tokens.
• Context Extension Survey - All context extension techniques surveyed.

Provider Docs

• Anthropic Long Context Tips - Place docs at top, use XML tags.
• Anthropic Context Windows - How context works, server-side compaction.
• Anthropic Context Engineering - Finding the smallest high-signal token set.
• Anthropic Long-Running Agents - Managing context across extended workflows.

Chunking & Splitting

• Pinecone Chunking Guide - Fixed-length, semantic, hierarchical.
• Advanced Chunking (Galileo) - Agentic and LLM-based.
• Context Engineering Guide - Curated papers and tools.
• Efficient Context Management (JetBrains) - Observation masking vs summarization.
• OpenProvence - Open reranker-pruner that drops ~99% off-topic sentences and compresses 80-90% of relevant RAG text; CPU-friendly.

KV Cache Optimization

Server-side optimizations for inference efficiency.

Inference Engines

• vLLM - PagedAttention, high-throughput inference.
• SGLang - RadixAttention for automatic KV cache reuse.
• GPUStack - GPU cluster manager for vLLM/SGLang.
• NVIDIA Dynamo - Datacenter-scale distributed inference with KV-cache-aware routing and disaggregated prefill/decode; ~2x faster TTFT, 7x throughput/GPU.
• llm-d - Kubernetes-native distributed serving with prefix-cache-aware routing and tiered KV offload to CPU/disk (3x output throughput).
• Mooncake - Distributed KVCache engine (the serving platform behind Moonshot AI's Kimi); integrated into vLLM for high-bandwidth KV-cache transfer and cross-instance prefix reuse across disaggregated prefill/decode.

Compression Tools

• NVIDIA kvpress - KV cache compression made easy.
• R-KV - Redundancy-aware compression (NeurIPS 2025).
• llm-compressor - Compression for deployment with vLLM.
• NVIDIA Model Optimizer - Quantization, pruning, distillation, speculative decoding.
• TurboQuant - Google's ICLR 2026; 5x KV cache compression.
• aibrix - Cost-efficient infrastructure for GenAI inference.
• LMCache - KV cache layer for vLLM/SGLang; offloads and reuses caches across engines (CPU/disk/S3) to cut TTFT for long-context, multi-turn, and RAG.
• kvcached - Virtualized elastic KV cache decoupling virtual/physical GPU memory for dynamic GPU sharing; plugs into vLLM/SGLang.
• KVzip - Query-agnostic KV cache eviction via context reconstruction (NeurIPS 2025 Oral); 3-4x memory reduction, 2x lower latency.
• DeepSpec - DeepSeek's open-source speculative decoding stack powering DSpark; 60-85% faster on V4-Flash, 57-78% on V4-Pro; outperforms Eagle-3. MIT license.

Research

• Speculative Sampling - Fast inference via speculative decoding.
• Awesome KV Cache Compression - Must-read paper list.

The full KV-cache paper table lives in the Academic Papers section below.

Educational

• mini-sglang - Learn LLM serving internals.
• tiny-llm - Build a tiny vLLM on Apple Silicon.

Browser Tool Efficiency

Different browser automation approaches consume vastly different context.

For 10-page workflows: WebFetch = ~15KB vs Playwright = ~330KB total context consumed.

Why Accessibility Trees Are Efficient

The accessibility tree strips visual styling to retain only semantic structure (name, role, state, value). 10-50x smaller than raw HTML. See: Token cost analysis in browser MCPs.

Further Reading

• WebFetch vs WebSearch analysis - Deep comparison.
• browser-use - Foundation library for AI browser agents.
• Chrome full accessibility tree - DevTools feature.
• mcp-compressor - MCP proxy that shows a compressed tool surface first and fetches full schemas on demand, cutting tokens spent on large MCP tool descriptions.

Cost Tracking Tools

• Langfuse - Open-source LLM observability + cost tracking. Cost tracking docs. Acquired by ClickHouse (Jan 2026); still actively developed, MIT-licensed.
• Helicone - LLM observability, 300+ models, SOC 2. Cost tracking cookbook. Acquired by Mintlify (Mar 2026); now maintenance-only — security/bug fixes and new-model support continue, no new feature work.
• LiteLLM Spend Tracking - Per-key/team spend tracking and budget routing for the LiteLLM proxy across 100+ LLMs.
• tokencost - USD cost estimates for 400+ LLMs.
• AgentOps - Agent monitoring with LLM cost tracking.
• agenttrace - Local-first TUI that reads Claude Code / Codex / Gemini / Aider / Cursor sessions to surface tokens, cost, cache use, retries, and latency.
• Future AGI traceAI - OpenTelemetry-based AI tracing capturing per-call tokens, cost, and latency across 35+ frameworks.
• ccusage - Fast local CLI reporting tokens and cost across 14+ coding agents (Claude Code, Codex, Gemini CLI, Copilot); offline, no upload.
• OpenLLMetry - OpenTelemetry-based GenAI observability instrumenting LLM and vector-DB calls with per-call token and latency telemetry.
• MLflow - Open-source AI/ML platform with GenAI observability in MLflow 3.x: LLM call tracing with per-span token tracking, prompt optimization tooling, and AI Gateway integration for cost control.
• Helicone AI Gateway - Fastest open-source AI gateway (Rust).
• Anthropic Token Counter - Free pre-flight token counting endpoint.
• tiktoken - OpenAI's fast BPE tokenizer (Python/Rust), 3-6x faster.
• LangSmith Cost Tracking - Automatic recording with dashboards.
• LlamaIndex Cost Analysis - Estimate costs before calls.

Pricing Comparison

Live Pricing Tools

• Price Per Token - Daily-updated, 300+ models.
• Artificial Analysis Calculator - Free calculator, 100+ models.
• Artificial Analysis Leaderboard - Quality + price + speed.
• Simon Willison's LLM Prices - Interactive calculator.
• Helicone LLM Cost Comparison - 300+ model calculator.
• Kingy.ai AI API Price Index - Versioned, source-audited prices for exact API offerings; preserves seller, route, region, service tier, billing mode, cache treatment, and context band, with CC BY JSON/CSV and SHA-256 manifests.
• CostGoat - 302+ APIs from 10+ providers.
• Langtail - Side-by-side comparison.
• WhatLLM - 256 models, 43+ providers, weekly updates.

Provider Pricing Pages

• Anthropic Pricing - Official Claude model pricing.
• OpenAI Pricing - Official OpenAI API pricing.
• Google Gemini Pricing - Official Gemini API pricing.
• DeepSeek Pricing - Official DeepSeek pricing.
• Mistral Pricing - Official Mistral pricing.

Notable Recent Pricing (June–July 2026)

Tokenizer note (Anthropic): Claude Opus 4.7+, Sonnet 5, and Fable 5 use a newer tokenizer that produces roughly 30% more tokens for the same text; per-token prices are unchanged, so the effective cost of a fixed input rises proportionally (Sonnet 4.6 and earlier keep the previous tokenizer). Benchmark your real workload before assuming a newer model lowers cost — confirmed on Anthropic's official pricing docs (already linked under Provider Pricing Pages above).

Prompt Engineering for Efficiency

Official Guides

• Anthropic Prompt Engineering - Master guide.
• Anthropic Claude 4 Best Practices - Model-specific.
• Anthropic Interactive Tutorial - 9-chapter course.
• Anthropic Tool Search - 85% token reduction for large tool libraries.
• OpenAI Prompt Engineering - Strategies and tactics.
• OpenAI Cost Optimization - Input minimization, model selection, caching.
• OpenAI Optimization Cookbook - Collection of notebooks.
• Token-Efficient Tool Use (Anthropic) - 70% output token reduction.

Community

• PromptingGuide: Optimizing - Compression, abstraction, filtering.
• Prompt Bloat Impact (MLOps) - Quality degrades with bloat.

Concise Reasoning Research

• Token Complexity - Each task has intrinsic minimum tokens for success.
• Verbosity != Veracity - Demystifying verbosity in LLM outputs.
• Incorporating Token Usage - Token usage as prompting strategy metric.

Comprehensive Guides

• 8 Strategies to Cut API Spend 80% (2026)
• Redis Token Optimization - Semantic caching, ~73% cost reduction.
• How I Reduced Token Costs by 90%
• LLM Token Optimization Strategies
• Monitor and Cut LLM Costs 90% (Helicone)
• LLM Caching Strategies (CostLens) - "90% savings most developers don't know about."
• AI Agent Cost Optimization - 60-70% of agent calls suit small models.
• Practical Cost + Latency Reduction
• Vantage LLM Cost Guide - Enterprise monitoring.
• Semantic Highlight for RAG (Zilliz) - 70-80% token reduction.
• Optimizing LLM in Production (Hugging Face) - Quantization, Flash Attention.
• Hugging Face Inference Optimization - Transformers library.
• Epoch AI: LLM Inference Price Trends - Data showing inference cost per fixed performance halving roughly every two months.
• ProjectDiscovery: Cut LLM Costs 59% With Caching - Raised cache hit rate from 7% to 84% across 9.8B cached tokens.
• Cockroach Labs: Agentic AI Costs at Scale - Re-sent context can be 62% of agent inference bills; mitigations.
• Together AI: Serving DeepSeek-V4 Long Context - Compressed KV layouts and eviction expanding single-node KV capacity 1.2M to 3.7M tokens.
• GitHub: Token Efficiency in Agentic Workflows - Pruning unused MCP tool schemas and swapping MCP calls for the gh CLI cut agentic-CI token spend 43-62% across real workflows.

Academic Papers

Prompt Compression

Model Routing & Cascading

Context & Inference

KV Cache & Inference

Prompt Optimization

Community Resources

Related Projects

• LLM Safe Haven - Security toolkit for AI coding agents. npx llm-safe-haven hardens Claude Code, Cursor, Windsurf in 60 seconds. Companion project — agent retries from security failures waste tokens.
• Awesome AI Efficiency (Pruna) - Curated list on making AI faster, cheaper, smaller, and greener.
• Awesome Efficient LLM - Large curated list of efficient-LLM papers and tools.

Blogs

• Simon Willison: LLM Pricing - Ongoing coverage of cost collapse.
• Simon Willison: LLMs in 2024 - MoE efficiency, cost trends.
• Eugene Yan: LLM Patterns - Caching (50%+ savings), fine-tuning, RAG, guardrails.
• Chip Huyen: AI OSS Analysis - 900 most popular AI tools analyzed.
• goodailist.com - Daily-updated tracker of 15K+ AI repos.

Discussions

• HN: How Are You Handling LLM API Costs in Production?
• HN: The LLM Agent Cost Curve
• HN: Genosis - LLM Cost Optimization

Podcasts

• Latent Space: Artificial Analysis - The "smiling curve of AI costs".

---

Licensed under CC BY 4.0. See LICENSE.