RAX Compute Gateway

One API. Every AI model.

RAX Compute Gateway is an open-source AI Compute Gateway. It gives applications one stable, OpenAI-compatible API for multiple model providers while centralizing routing, authentication, retries, limits, and telemetry.

Application -> RAX Compute Gateway -> OpenAI | Anthropic | Gemini | future providers

Status

RAX Compute Gateway v0.3.0 adds bounded Agent compatibility: OpenAI-style function tools, streamed tool-call deltas, tool-result messages, structured outputs, capability-safe routing, and per-key tool permission. The prior v0.2.0 release added the operator console, one-command self-hosted quickstart, and the public, abuse-resistant hosted evaluation described below. Release artifacts include a signed multi-architecture image, Helm chart, OpenAPI contract, checksums, SBOM, and provenance. The documents in this repository remain normative unless an accepted Architecture Decision Record (ADR) supersedes them.

Hosted-service status: the public RAX deployment is currently paused between scheduled evaluation windows to control operating costs. The hosted API, operator console, and five-minute demo may therefore be unavailable. To request a temporary evaluation window, open a GitHub Issue and wait for confirmation before trying the hosted URLs. Never include API keys, provider credentials, or other secrets in an issue.

MVP capabilities

• POST /v1/chat/completions, including streaming
• OpenAI, Anthropic, and Gemini adapters
• Agent function calling across all three adapters, including streaming
• json_object and JSON Schema structured-output capability routing
• Stable public model aliases and explicit provider models
• Deterministic routing, fallback, timeout, and retry policies
• RAX Compute Gateway API keys and bring-your-own-provider-key (BYOK) operation
• PostgreSQL metadata, optional Redis coordination, and stateless gateway nodes
• OpenTelemetry traces and metrics plus structured, redacted logs
• Docker Compose for local use and Kubernetes manifests/Helm for production
• TypeScript and Python SDKs generated from the public contract

Non-goals include gateway-side tool execution, the OpenAI Responses API, a hosted billing system, a marketplace, GPU scheduling, multimodal generation, and autonomous quality-based model selection. Post-MVP additions are narrowly scoped by accepted ADRs.

Use with AI Agents (v0.3)

Any Agent or harness that can use the OpenAI Chat Completions protocol can target RAX by changing its base URL, API key, and model. Use rax/agent so routing is restricted to tool-capable candidates:

Self-hosted base URL: http://localhost:8080/v1
Hosted base URL (when an evaluation window is active): https://api.rax-digital.com/v1
API Key:  your RAX API key
Model:    rax/agent

The key must have Allow Agent tool calls enabled. The Agent—not the gateway—executes tools and returns their results. Existing text-only and hosted demo keys remain denied by default.

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $RCG_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "rax/agent",
    "messages": [{"role":"user","content":"What is the weather in Tokyo?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"],
          "additionalProperties": false
        }
      }
    }],
    "tool_choice": "auto"
  }'

See Agent and harness integration for the supported boundary, Hermes configuration, multi-turn examples, and framework compatibility notes.

Choose your first run

60-second hosted trial (available by request)

The hosted deployment is currently stopped between scheduled evaluation windows, so api.rax-digital.com/demo will not respond while the service is paused. To try it, open a GitHub Issue requesting a temporary evaluation window. After the operator confirms that the service is online, open the demo, verify with GitHub, and copy the generated curl. Do not post credentials or claimed API keys in the issue.

When active, the complete public path is: claim a key, run the generated command, and receive a normalized model response in about 60 seconds.

The service issues a unique API key that expires after five minutes; there is no shared public key in this repository. The trial is intentionally limited to one low-cost model, non-streaming calls, small input/output budgets, one concurrent request, per-account cooldown, and a global daily budget. See hosted demo design and operation.

The v0.3.0 experience page also introduces the rax/agent integration path. For safety, the claimed public key remains text-only; use an operator-issued key with Allow Agent tool calls enabled, or self-host, for function tools.

The hosted service is for evaluation only. Applications should self-host or obtain a normal customer key rather than depend on trial availability.

5–10 minute self-hosted start

Prerequisites: Docker 26+ with Docker Compose, curl, and an OpenAI API key for the first request. A fork is needed only when contributing code; to try the gateway, clone the upstream repository and run one command:

git clone https://github.com/radium0090/Compute-Gateway.git
cd Compute-Gateway
sh scripts/quickstart.sh

The script creates a private .env, generates local gateway secrets, prompts for the provider key without echoing it, starts PostgreSQL, Redis, telemetry, and the gateway, provisions a local client key, and prints the first model response. It never sends the OpenAI key anywhere except the configured OpenAI endpoint. Stop the stack with docker compose down.

A successful run ends with a normalized JSON chat response and a local rcg_dev_... credential shown once. If the provider rejects the final request, the local stack remains running for diagnosis; check provider billing/model access, update .env, and rerun the script.

To understand or run each operation manually instead:

cp .env.example .env
# Replace the fake RCG secrets and OPENAI_API_KEY in .env.
docker compose up --build --wait
docker compose exec postgres psql -U rcg -d compute_gateway -c \
  "INSERT INTO tenants (id, name, status) VALUES ('123e4567-e89b-42d3-a456-426614174000', 'local', 'active') ON CONFLICT DO NOTHING"
RCG_API_KEY="$(docker compose run --rm gateway keys create \
  --tenant-id 123e4567-e89b-42d3-a456-426614174000 \
  --name local-app --environment dev --models 'rax/*' --allow-streaming)"
export RCG_API_KEY
test -n "$RCG_API_KEY"

The key command emits the new credential once; command substitution keeps it out of the terminal and stores it in the current shell. Keep it out of shell history, source control, logs, and URLs.

Compose supplies PostgreSQL and Redis. Outside production, running the gateway without RCG_REDIS_URL uses process-local limits and circuit state; every production replica requires Redis coordination and fails startup if it is not configured.

Send a request:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $RCG_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "rax/fast",
    "messages": [{"role": "user", "content": "Hello from RAX Compute Gateway"}]
  }'

RAX Compute Gateway accepts the OpenAI client by changing its base URL:

import os

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["RCG_API_KEY"],
    base_url="http://localhost:8080/v1",
)

response = client.chat.completions.create(
    model="rax/fast",
    messages=[{"role": "user", "content": "Hello"}],
)

Operator console (v0.2)

Set RCG_ADMIN_ENABLED=true, configure the exact RCG_ADMIN_ORIGIN, and use a dedicated RCG_ADMIN_SESSION_PEPPER. After migrations, create the first administrator with a temporary password supplied on standard input:

printf '%s\n' "$RCG_ADMIN_TEMPORARY_PASSWORD" | docker compose run --rm -T gateway \
  admins create --email owner@example.com --display-name 'Gateway Owner'

Open http://localhost:8080/admin/. The temporary password must be replaced at first login. The console manages tenants and one-time-display API keys and shows bounded service/activity metadata. It never exposes provider credentials, password/session hashes, API-key hashes, prompts, or completions.

Documentation

Runnable curl, Node.js, and Python examples are included for a locally running gateway.

Product and implementation

• Vision
• MVP specification
• Architecture
• Repository layout
• Roadmap
• Implementation status and known gaps

API and runtime behavior

• API contract
• OpenAPI workflow
• Routing
• Provider adapters
• Authentication
• SDKs
• Agent and harness integration
• Configuration

Operations

• Deployment
• Docker
• Kubernetes
• Database
• Security
• Observability
• CI/CD
• GitHub Actions
• Release-candidate runbook
• v0.1.0 RC evidence template
• Rollback runbook
• Incident response
• Security reporting

Project governance

• Code of Conduct
• Contributing
• Coding standards
• Licensing
• FAQ
• Architecture decisions
• Changelog

Development contract

The keywords MUST, MUST NOT, SHOULD, and MAY are used as defined by RFC 2119. Implementation work should follow this precedence order:

1. Accepted ADRs
2. API and security contracts
3. MVP acceptance criteria
4. Architecture and operational guidance
5. Roadmap ideas

If documents conflict, open an issue and resolve it with an ADR before merging behavior that changes the public contract.

Community

Use GitHub Issues for bugs and scoped features, Discussions for questions and design exploration, and pull requests for reviewed changes. Please read CONTRIBUTING before submitting code.

You do not need to fork merely to run the gateway. To contribute, first fork the repository, clone your fork, create a feature branch, and open a pull request as described in the contribution guide.

License

The project is licensed under Apache License 2.0. See licensing.md for dependency and contribution rules.