AI Gateway

A self-hosted, cost-efficient AI infrastructure platform that provides:

• Speech-to-Speech Translation — Real-time multilingual audio translation via STT → LLM → TTS pipeline
• GPU Deployment & Management — Multi-provider cascade deploy (RunPod → Vast.ai → TensorDock → Modal) with auto-scaling, idle management, and crash recovery
• AI Provider Abstraction — Unified interface over Groq, OpenAI, Fireworks, OpenRouter, Ollama, Deepgram, and self-hosted GPUs
• Hybrid Routing — Race GPU vs cloud, auto-select cheapest, latency-based demotion, canary deployments
• Observability — Request logging, provider metrics, latency tracking, circuit breakers, Prometheus metrics

Built with TypeScript, providing a unified interface for AI providers and multi-tier GPU autoscaling with automatic failover.

Features

• Transparent Speech API — Single POST /v1/speech endpoint handles STT → LLM → TTS. Transport (GPU vs cloud) is hidden from the caller. No SSE/WebSocket/WebRTC in client code.
• Deployments (Scaleway) — PUT /v1/deployments/:name with any Docker image (or a profile like qwen3-tts) → autoscaled replicas, scale to zero, cold-start wait, per-deployment replica bounds via API. See docs/deployments.md
• Multi-tier GPU Autoscaler — Cascade through GPU providers (RunPod, TensorDock, Modal) with automatic failover, health checking, idle watchdog, and cost monitoring
• AI Provider Abstraction — Unified interface for 8+ providers (OpenAI, Groq, Fireworks, OpenRouter, Modal, self-hosted) across STT, TTS, LLM, Image, and Realtime modalities
• Provider Fallback Chains — Declarative, config-driven fallback with cooldown and credit exhaustion tracking
• Predictive Warmup — ML-based usage prediction to pre-boot GPUs before demand spikes
• Spend Tracking — Per-request cost estimation, budget alerts, and provider spend monitoring
• Infrastructure Utilities — SkyPilot CLI integration, SSH/SCP helpers

Installation

# Workspace dependency (in monorepo)
bun add @parle/ai-gateway@workspace:*

Quick Start

Pipeline API (Recommended)

Send audio, get back transcription + translation + TTS audio in one JSON response. The gateway handles GPU vs cloud routing transparently.

# Full pipeline: audio in → JSON out
curl -X POST "http://localhost:4000/v1/speech?source=fr&target=en&speaker=Ryan" \
  --data-binary @audio.wav -H "Content-Type: audio/wav"

# Response:
# {
#   "transcription": "Bonjour le monde",
#   "response": "Hello world",
#   "audio_base64": "UklGR...",
#   "content_type": "audio/wav",
#   "timing": { "total_ms": 1234, "used_gpu": true }
# }

AIClient (TypeScript)

import { createAIClient } from '@ai-gateway/client';

const client = createAIClient({ registry, defaultProfile });

// Full pipeline — tries GPU first, falls back to cloud automatically
const result = await client.pipeline(audioBuffer, systemPrompt);
// result.stt.text       → "Bonjour le monde"
// result.chat.content   → "Hello world"
// result.tts.audio      → Buffer (WAV)
// result.usedGpu        → true/false
// result.totalLatencyMs → 1234

// Individual stages (also transport-transparent)
const stt  = await client.transcribe(audioBuffer);   // { text, latencyMs }
const chat = await client.chat(messages);             // { content, latencyMs }
const tts  = await client.synthesize(text);           // { audio, contentType }

Transport Policy

SSE, WebSocket, and WebRTC are blocked at the proxy level (return 410 Gone). All client code uses the JSON pipeline endpoint or the AIClient API. This keeps transport concerns inside the gateway — callers never parse SSE events, manage WebSocket frames, or negotiate WebRTC.

Gateway Factory (Autoscaler)

import { createGateway } from '@ai-gateway';

const gateway = createGateway({
  storage: myStorageAdapter,  // implements GatewayStorage
  stateStore: redisAdapter,   // optional, falls back to in-memory
  hooks: {                    // optional observability
    onScaleUp: (e) => console.log('GPU booting:', e),
    onCostAlert: (e) => console.log('Budget alert:', e),
  },
});

const status = await gateway.handleGet(userId);
const result = await gateway.handleAction(userId, 'boot', { tier: 0 });

AI Providers

import { groqLLM, openaiSTT, openaiTTS } from '@ai-gateway/providers';

// Individual providers
const stt = openaiSTT({ apiKey: '...' });
const llm = groqLLM({ apiKey: '...' });
const tts = openaiTTS({ apiKey: '...' });

const transcript = await stt.transcribe({ audio: audioBuffer });
const response = await llm.chat({ messages: [{ role: 'user', content: transcript.text }] });
const audio = await tts.synthesize({ text: response.text, voice: 'alloy' });

Provider Fallback

import { withProviderFallback } from '@ai-gateway/providers';

const result = await withProviderFallback(
  [groqLLM({ apiKey: '...' }), openaiLLM({ apiKey: '...' })],
  (provider) => provider.chat({ messages }),
  { maxRetries: 2, cooldownMs: 30_000 }
);

GPU Provider Clients

import { RunPodClient, TensorDockClient } from '@ai-gateway/gpu-providers';

const runpod = new RunPodClient({ apiKey: '...' });
const pod = await runpod.boot({
  gpuType: 'RTX A5000',
  dockerImage: 'marcosremar/parle-s2s:latest',
  ports: ['8000/http', '22/tcp'],
});

await runpod.waitForReady(pod.id);
await runpod.terminate(pod.id);

Module Exports

Architecture

                    ┌─────────────────┐
                    │  createGateway  │  ← Single entry point
                    └────────┬────────┘
                             │
            ┌────────────────┼────────────────┐
            │                │                │
    ┌───────▼──────┐  ┌─────▼─────┐  ┌───────▼──────┐
    │  Autoscaler  │  │ Providers │  │   Handlers   │
    │   Engine     │  │  Registry │  │  (Next.js)   │
    └───────┬──────┘  └─────┬─────┘  └──────────────┘
            │               │
   ┌────────┼────────┐     │
   │        │        │     │
┌──▼──┐ ┌──▼──┐ ┌───▼─┐  ┌▼────────────┐
│RunPod│ │TD   │ │Modal│  │OpenAI, Groq,│
│      │ │     │ │     │  │Fireworks,...│
└──────┘ └─────┘ └─────┘  └─────────────┘

Design Principles

• Framework-agnostic — No hard dependencies on Prisma, Redis, or Next.js. All external dependencies injected via interfaces.
• Modular — Each subpath is independently importable. Browser SDK has zero Node.js dependencies.
• Tier cascade — GPU autoscaler walks through tiers (cheapest first) with automatic failover to the next tier on failure.
• Provider fallback — AI providers fail over automatically with cooldown tracking and credit exhaustion detection.

Supported Providers

AI Providers

GPU Cloud Providers

Providers to explore — Firecracker / gVisor / Kata GPU sandboxing

These offer sandboxed GPU runtimes (sub-second cold start), per-second billing, and fit the Modal-style scale-up model we could adopt instead of rolling our own. Priority in order of fit:

Measured comparison baseline (Hyperstack L40 + our offloadOnIdle): 1.3s wake on the same VM. Competitive with Northflank (1–2s). Beaten by RunPod Serverless FlashBoot (<200ms). Koyeb GPU currently does not deliver its 200ms CPU story for GPU workloads (VRAM snapshot limitation).

Deeper notes per provider

• Beam / Beta9 (GitHub) — the only serverless GPU runtime we've found that is open source (AGPL-3.0), supports runtime choice (runc / gVisor), uses cuda-checkpoint for GPU snapshots, and has a CLIP lazy-loading image format (weights fault in on-demand from S3/FUSE — no upfront image pull). Self-hostable, so you could run Beta9 directly on Hyperstack L40 VMs and get Modal-class primitives without the Modal bill. Managed SaaS cold starts: 2–3s typical, 50ms warm.
• Koyeb — Firecracker + Cloud Hypervisor (for GPU VFIO passthrough). Their "200ms wake via eBPF + Light Sleep snapshots" only applies to CPU workloads; for GPU, VRAM snapshot preservation is explicitly still a work-in-progress on their end.
• Northflank — unique in letting the caller choose runtime per workload (Firecracker / Kata / gVisor). End-to-end sandbox create ~1–2s including image pull. No L40/L40S in catalog; A100 40G at $1.42/hr is the closest sweet-spot.
• RunPod Serverless FlashBoot — closest real competitor to Modal on cold start (<200ms in 48% of calls per their data). Proprietary runtime, per-second billing. Best ROI if you don't need sandbox choice.
• Inferless — NOT a microVM runtime. Just hf-transfer accelerated weight download + container. Measured cold starts: 7.8s (Phi-3, T4), 13.3s (Llama-3 8B). Fine for small models on T4/A10, doesn't replace Hyperstack offload.
• Salad.com — distributed-compute marketplace (home PCs), 60k+ GPUs from $0.02/hr. No sandbox isolation, no SLA — batch only.

Sources: Koyeb pricing, Koyeb scale-to-zero blog, Koyeb GPU scale-to-zero caveats, Northflank pricing, Northflank sandbox runtimes, RunPod Serverless, Beam Beta9 GitHub, Beam serverless platform guide, Inferless pricing, Salad pricing.

Automated AI Quality Controls

This section is generated from docs/quality-controls.json. Run bun run quality:readme after changing the quality gate manifest.

Local Quality Commands

bun run quality:ai              # typecheck + lint + build + fitness + supply-chain + architecture
bun run quality:ai:test         # quality:ai plus the unit suite
bun run quality:ai:deep         # quality:ai:test plus mutation testing
bun run quality:fitness:debt    # inspect current complexity/module-size structural debt
bun run quality:fitness:strict  # fail on current complexity/module-size ratchet warnings
bun run quality:supply-chain    # audit CI/Docker supply-chain hardening
bun run quality:architecture:strict # include circular dependency warnings
bun run quality:deadcode        # Knip dead-code/dependency audit without failing on existing debt
bun run quality:deadcode:strict # fail on Knip issues after the debt is triaged
bun run test:properties         # property-based invariant tests

Build

bun run build   # tsup → dist/ (ESM + CJS + declarations + sourcemaps)

Publishing to production

bun run deploy:prod             # from a clean checkout of origin/main

Production is a single Railway instance, so a publish restarts it. The script refuses during a parle class (the declared warmSchedule, Mon–Thu 17:30–20:30 Europe/Paris; --during-class overrides it for an emergency), a dirty tree or a HEAD that is not origin/main; then it stamps the build, runs railway up and waits until /health reports the new commit (plus fallbackProbe with AI_GATEWAY_ADMIN_KEY set). Never call railway up by hand.

Tech Stack

• TypeScript — Strict mode, ES2017 target
• Zod — Runtime config validation
• tsup — Bundler (ESM + CJS dual output)
• OpenAI SDK — Base client for OpenAI-compatible APIs

STT Quality Research — Whisper initial_prompt

The gateway's STT pipeline uses Whisper's initial_prompt parameter to anchor transcription to domain-specific vocabulary. The following research informs our implementation:

Key Techniques

Guidelines

• 224-token limit (~170 words) — keep prompts compact
• Language must match audio — wrong-language prompts degrade WER by ~19% (arXiv 2406.05806)
• Sentence-style prompts outperform keyword lists — use natural flowing prose, not comma-separated terms (arXiv 2406.05806)
• Vocabulary-dense prompts — include speaker name, technical terms, and domain proper nouns; generic/register-only prompts show no gain (arXiv 2406.05806, confirmed in our benchmarks)
• Avoid verbatim transcript overlap — if prompt contains words that appear in the talk, Whisper treats them as "already transcribed" and may miss them in actual audio (observed in our benchmarks)
• Most recent tokens get highest attention weight — put the most relevant context last
• Static prompts plateau quickly — rolling context + LLM expansion gives the best combined result

Benchmark Results — TEDx French (4 videos, Groq Whisper-large-v3-turbo)

Ground truth: professional human subtitles (TEDx manual captions). WER-sem = WER after LLM normalization (removes style differences: numbers, abbreviations, punctuation).

Key findings:

1. Rolling context alone is the most impactful technique. Without it, WER is ~5pp higher on average. It resolves cross-boundary phrases and maintains spelling consistency chunk-to-chunk.

2. Seed context (LLM-expanded title) helps domain-specific talks significantly — Psychology (−8.4pp) and Philosophy (−6.0pp) benefit because the prompt anchors domain vocabulary (e.g. biais cognitif, heuristique, Schopenhauer).

3. Seed context does NOT help — or hurts — general-vocabulary talks (Klein, Fictions). Two mechanisms:

   • Vocabulary overlap: if the prompt contains words that appear verbatim in the talk, Whisper treats them as "already transcribed" and may miss or distort them in the actual audio.
   • Floor effect: talks already well-transcribed (WER ~15%) have little room to improve; any prompt competes with the rolling context for the 224-token window.

4. Prompt style matters: vocabulary-dense prompts outperform register-only prompts for technical content. Register-only prompts ("Conférence TEDx en français...") are neutral at best.

5. Raw WER increases slightly with seed prompts (~1-2pp) even when WER-sem improves — the prompt shifts Whisper's output style (more formal, different punctuation), which doesn't match the reference literally but is semantically correct.

Practical rule for the "Session title" field:

• Fill it for medical, legal, academic, or technical meetings → significant gains
• Leave it empty for general conversation, casual talks, or pop-culture content → rolling context is sufficient

References

• arXiv 2602.18966 — "Improved Domain-Specific ASR via Large Language Model Prompt Engineering" (NBA commentary). LLM-generated compact domain prompts: 17% relative WER reduction vs no prompt.
• arXiv 2406.05806 — "Do Prompts Really Prompt? Rethinking the Role of Prompts in Whisper Transcription". Comprehensive study of prompt strategies; sentence-style > keyword lists; language mismatch = catastrophic degradation.
• arXiv 2502.11572 — "Improving Whisper's Recognition of Rare Words via Initial Prompt Injection". Injection of rare/domain-specific terms in initial_prompt significantly improves recognition of OOV vocabulary.
• OpenAI Cookbook — Whisper Prompting Guide — Practical guide covering spelling correction, filler word suppression, punctuation style, and fictional context prompts.

License

Private — Part of the Parle ecosystem.