from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="your-proxy-api-key"
)

response = client.chat.completions.create(
    model="gemini/gemini-2.5-flash",  # provider/model format
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

curl -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-proxy-api-key" \
  -d '{
    "model": "gemini/gemini-2.5-flash",
    "messages": [{"role": "user", "content": "What is the capital of France?"}]
  }'

1. Go to API Settings
2. Select "Proxy" or "Custom OpenAI" mode
3. Configure:
   • API URL: http://127.0.0.1:8000/v1
   • API Key: Your PROXY_API_KEY
   • Model: provider/model_name (e.g., gemini/gemini-2.5-flash)
4. Save and start chatting

In your configuration file (e.g., config.json):

{
  "models": [
    {
      "title": "Gemini via Proxy",
      "provider": "openai",
      "model": "gemini/gemini-2.5-flash",
      "apiBase": "http://127.0.0.1:8000/v1",
      "apiKey": "your-proxy-api-key"
    }
  ]
}

Claude Code natively supports custom Anthropic API endpoints. The recommended setup is to edit your Claude Code settings.json:

{
  "env": {
    "ANTHROPIC_AUTH_TOKEN": "your-proxy-api-key",
    "ANTHROPIC_BASE_URL": "http://127.0.0.1:8000",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "gemini/gemini-3-pro",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "gemini/gemini-3-flash",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "openai/gpt-5-mini"
  }
}

Now you can use Claude Code with Gemini, OpenAI, or any other configured provider.

from anthropic import Anthropic

client = Anthropic(
    base_url="http://127.0.0.1:8000",
    api_key="your-proxy-api-key"
)

# Use any provider through Anthropic's API format
response = client.messages.create(
    model="gemini/gemini-3-flash",  # provider/model format
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.content[0].text)

• Global timeout with deadline-driven retries
• Escalating cooldowns per model (10s → 30s → 60s → 120s)
• Key-level lockouts for consistently failing keys
• Stream error detection and graceful recovery
• Batch embedding aggregation for improved throughput
• Automatic daily resets for cooldowns and usage stats

• Auto-discovery of API keys from environment variables
• OAuth discovery from standard paths (~/.gemini/)
• Duplicate detection warns when same account added multiple times
• Credential prioritization — paid tier used before free tier
• Stateless deployment — export OAuth to environment variables
• Local-first storage — credentials isolated in oauth_creds/ directory

• Model whitelists/blacklists with wildcard support
• Per-provider concurrency controls (OPTIMAL_CONCURRENT_REQUESTS_PER_KEY_<PROVIDER> and MAX_CONCURRENT_REQUESTS_PER_KEY_<PROVIDER>)
• Rotation modes — balanced (distribute load) or sequential (use until exhausted)
• Priority multipliers — higher concurrency for paid credentials
• Model quota groups — shared cooldowns for related models
• Temperature override — prevent tool hallucination issues
• Weighted random rotation — unpredictable selection patterns

Gemini CLI:

• Zero-config Google Cloud project discovery
• Internal API access with higher rate limits
• Automatic fallback to preview models on rate limit
• Paid vs free tier detection

NVIDIA NIM:

• Dynamic model discovery
• DeepSeek thinking support

• Per-request file logging with --enable-request-logging
• Raw I/O logging with --enable-raw-logging (proxy boundary payloads)
• Unique request directories with full transaction details
• Streaming chunk capture for debugging
• Performance metadata (duration, tokens, model used)
• Provider-specific logs for active custom providers

Proxy Settings

Per-Provider Settings

Advanced Features

Control which models are exposed through your proxy.

Blacklist Only

# Hide all preview models
IGNORE_MODELS_OPENAI="*-preview*"

Pure Whitelist Mode

# Block all, then allow specific models
IGNORE_MODELS_GEMINI="*"
WHITELIST_MODELS_GEMINI="gemini-2.5-pro,gemini-2.5-flash"

Exemption Mode

# Block preview models, but allow one specific preview
IGNORE_MODELS_OPENAI="*-preview*"
WHITELIST_MODELS_OPENAI="gpt-4o-2024-08-06-preview"

Logic order: Whitelist check → Blacklist check → Default allow

Concurrency Limits

# Balanced mode defaults to optimal=1 and max=-1, so it spreads first
# but will stack on busy keys instead of blocking when every key is busy.
ROTATION_MODE_OPENAI=balanced
OPTIMAL_CONCURRENT_REQUESTS_PER_KEY_OPENAI=1
MAX_CONCURRENT_REQUESTS_PER_KEY_OPENAI=-1

# Sequential mode defaults to optimal=-1 and max=-1 for sticky/unlimited use.
ROTATION_MODE_GEMINI=sequential
OPTIMAL_CONCURRENT_REQUESTS_PER_KEY_GEMINI=-1
MAX_CONCURRENT_REQUESTS_PER_KEY_GEMINI=-1

# Constrained providers can set optimal and max to the same value.
MAX_CONCURRENT_REQUESTS_PER_KEY_GEMINI_CLI=1
OPTIMAL_CONCURRENT_REQUESTS_PER_KEY_GEMINI_CLI=1

# Mode-specific forms override provider-wide values only for that mode.
MAX_CONCURRENT_REQUESTS_PER_KEY_OPENAI_BALANCED=-1
OPTIMAL_CONCURRENT_REQUESTS_PER_KEY_OPENAI_BALANCED=1

optimal is a soft target used for capacity phases: the rotator prefers credentials below optimal, then stacks on healthy credentials when no below-optimal credential remains. max is the hard safety ceiling; 0 or any negative value means unlimited.

Rotation Modes

# sequential (default): Use one key until it errors/exhausts, preserving provider-side cache locality
ROTATION_MODE_GEMINI=sequential

# balanced: Distribute load evenly - opt in for per-minute rate limits
ROTATION_MODE_OPENAI=balanced

Priority Multipliers

Paid credentials can handle more concurrent requests. Legacy priority multipliers apply to hard max concurrency; provider-specific optimal multipliers can also raise the soft target where supported:

# Priority 1: 10x concurrency
CONCURRENCY_MULTIPLIER_GEMINI_CLI_PRIORITY_1=10

# Priority 2: 3x
CONCURRENCY_MULTIPLIER_GEMINI_CLI_PRIORITY_2=3

Model Quota Groups

Models sharing quota limits:

# Example: group provider models that share quota
QUOTA_GROUPS_GEMINI_CLI_PRO="gemini-2.5-pro,gemini-3-pro-preview"

Fine-grained control over HTTP timeouts:

TIMEOUT_CONNECT=30              # Connection establishment
TIMEOUT_WRITE=30                # Request body send
TIMEOUT_POOL=60                 # Connection pool acquisition
TIMEOUT_READ_STREAMING=180      # Between streaming chunks (3 min)
TIMEOUT_READ_NON_STREAMING=600  # Full response wait (10 min)

Recommendations:

• Long thinking tasks: Increase TIMEOUT_READ_STREAMING to 300-360s
• Unstable network: Increase TIMEOUT_CONNECT to 60s
• Large outputs: Increase TIMEOUT_READ_NON_STREAMING to 900s+

Uses Google OAuth to access internal Gemini endpoints with higher rate limits.

Setup:

1. Run python -m rotator_library.credential_tool
2. Select "Add OAuth Credential" → "Gemini CLI"
3. Complete browser authentication
4. Credentials saved to oauth_creds/gemini_cli_oauth_1.json

Features:

• Zero-config project discovery
• Automatic free-tier project onboarding
• Paid vs free tier detection
• Smart fallback on rate limits
• Quota baseline tracking with background refresh (accurate remaining quota estimates)
• Sequential rotation mode (uses credentials until quota exhausted)

Quota Groups: Models that share quota are automatically grouped:

• Pro: gemini-2.5-pro, gemini-3-pro-preview
• 2.5-Flash: gemini-2.0-flash, gemini-2.5-flash, gemini-2.5-flash-lite
• 3-Flash: gemini-3-flash-preview

All models in a group deplete the shared quota equally. 24-hour per-model quota windows.

Environment Variables (for stateless deployment):

Single credential (legacy):

GEMINI_CLI_ACCESS_TOKEN="ya29.your-access-token"
GEMINI_CLI_REFRESH_TOKEN="1//your-refresh-token"
GEMINI_CLI_EXPIRY_DATE="1234567890000"
GEMINI_CLI_EMAIL="[email protected]"
GEMINI_CLI_PROJECT_ID="your-gcp-project-id"  # Optional
GEMINI_CLI_TIER="standard-tier"  # Optional: standard-tier or free-tier

Multiple credentials (use _N_ suffix where N is 1, 2, 3...):

GEMINI_CLI_1_ACCESS_TOKEN="ya29.first-token"
GEMINI_CLI_1_REFRESH_TOKEN="1//first-refresh"
GEMINI_CLI_1_EXPIRY_DATE="1234567890000"
GEMINI_CLI_1_EMAIL="[email protected]"
GEMINI_CLI_1_PROJECT_ID="project-1"
GEMINI_CLI_1_TIER="standard-tier"

GEMINI_CLI_2_ACCESS_TOKEN="ya29.second-token"
GEMINI_CLI_2_REFRESH_TOKEN="1//second-refresh"
GEMINI_CLI_2_EXPIRY_DATE="1234567890000"
GEMINI_CLI_2_EMAIL="[email protected]"
GEMINI_CLI_2_PROJECT_ID="project-2"
GEMINI_CLI_2_TIER="free-tier"

Feature Toggles:

GEMINI_CLI_QUOTA_REFRESH_INTERVAL=300  # Quota refresh interval in seconds (default: 300 = 5 min)

For platforms without file persistence (Railway, Render, Vercel):

1. Set up credentials locally:

   python -m rotator_library.credential_tool
   # Complete OAuth flows
   

2. Export to environment variables:

   python -m rotator_library.credential_tool
   # Select "Export [Provider] to .env"
   

3. Copy generated variables to your platform: The tool creates files like gemini_cli_credential_1.env containing all necessary variables.

4. Set SKIP_OAUTH_INIT_CHECK=true to skip interactive validation on startup.

Customize OAuth callback ports if defaults conflict:

python src/proxy_app/main.py [OPTIONS]

Options:
  --host TEXT                Host to bind (default: 0.0.0.0)
  --port INTEGER             Port to run on (default: 8000)
  --enable-request-logging   Enable detailed per-request logging
  --enable-raw-logging       Capture raw proxy I/O payloads
  --add-credential           Launch interactive credential setup tool

Examples:

# Run on custom port
python src/proxy_app/main.py --host 127.0.0.1 --port 9000

# Run with logging
python src/proxy_app/main.py --enable-request-logging

# Run with raw I/O logging
python src/proxy_app/main.py --enable-raw-logging

# Add credentials without starting proxy
python src/proxy_app/main.py --add-credential

See the Deployment Guide for complete instructions.

Quick Setup:

1. Fork the repository
2. Create a .env file with your credentials
3. Create a new Web Service pointing to your repo
4. Set build command: pip install -r requirements.txt
5. Set start command: uvicorn src.proxy_app.main:app --host 0.0.0.0 --port $PORT
6. Upload .env as a secret file

OAuth Credentials: Export OAuth credentials to environment variables using the credential tool, then add them to your platform's environment settings.

The proxy is available as a multi-architecture Docker image (amd64/arm64) from GitHub Container Registry.

Quick Start with Docker Compose:

# 1. Create your .env file with PROXY_API_KEY and provider keys
cp .env.example .env
nano .env

# 2. Create usage directory (usage_*.json files are created automatically)
mkdir usage

# 3. Start the proxy
docker compose up -d

# 4. Check logs
docker compose logs -f

Important: Create the usage/ directory before running Docker Compose so usage stats persist on the host.

Manual Docker Run:

# Create usage directory if it doesn't exist
mkdir usage

docker run -d \
  --name llm-api-proxy \
  --restart unless-stopped \
  -p 8000:8000 \
  -v $(pwd)/.env:/app/.env:ro \
  -v $(pwd)/oauth_creds:/app/oauth_creds \
  -v $(pwd)/logs:/app/logs \
  -v $(pwd)/usage:/app/usage \
  -e SKIP_OAUTH_INIT_CHECK=true \
  -e PYTHONUNBUFFERED=1 \
  ghcr.io/mirrowel/llm-api-key-proxy:latest

Development with Local Build:

# Build and run locally
docker compose -f docker-compose.dev.yml up -d --build

Volume Mounts:

Image Tags:

OAuth with Docker:

For OAuth providers such as Gemini CLI, you must authenticate locally first:

1. Run python -m rotator_library.credential_tool on your local machine
2. Complete OAuth flows in browser
3. Either:
   • Mount oauth_creds/ directory to container, or
   • Export credentials to .env using the export option

Option 1: Authenticate locally, deploy credentials

1. Complete OAuth flows on your local machine
2. Export to environment variables
3. Deploy .env to your server

Option 2: SSH Port Forwarding

# Forward callback ports through SSH
ssh -L 51121:localhost:51121 -L 8085:localhost:8085 user@your-vps

# Then run credential tool on the VPS

Systemd Service:

[Unit]
Description=LLM API Key Proxy
After=network.target

[Service]
Type=simple
WorkingDirectory=/path/to/LLM-API-Key-Proxy
ExecStart=/path/to/python -m uvicorn src.proxy_app.main:app --host 0.0.0.0 --port 8000
Restart=always

[Install]
WantedBy=multi-user.target

See VPS Deployment for complete guide.