UnlimitedNIM

Make NVIDIA's 40 RPM free tier feel unlimited. Smart priority queue proxy for coding agents.

Works with Cline · Roo Code · OpenCode · Any OpenAI-compatible agent

The Problem

NVIDIA's free NIM tier limits you to 40 requests per minute. Coding agents like Cline, Roo Code, and OpenCode generate bursts of traffic — an agent loop firing tool calls, context reads, and model invocations in parallel will slam straight into that wall. The result: a flood of 429 Rate Limit Exceeded errors, retried work, frustrated agents, and stalls in the middle of a task.

You don't get more throughput from NVIDIA — but you can stop wasting the window you have.

What This Is

UnlimitedNIM is a thin, streaming reverse proxy that sits between your coding agent and NVIDIA's API. It enforces NVIDIA's 40 RPM limit for you — in a way that's invisible to your agent:

• Sliding-window rate limiter — never fires more than 40 requests per minute, so NVIDIA never sees a burst and never returns 429.
• Priority queue — when the window is full, requests wait in line instead of failing:
  • priority 1 — user messages (you typed it, you're waiting)
  • priority 3 — agent loop tool calls (the bulk of traffic)
  • priority 5 — debug dumps and huge stack traces (fire when there's room)
• Full-duplex streaming — SSE responses stream straight through, chunk by chunk.
• Starvation protection — a low-priority request stuck longer than the TTL is promoted so it eventually fires.
• Backpressure — a bounded queue rejects with 503 + Retry-After instead of leaking memory or flooding NVIDIA.

The Screenshot-Worthy Stat

260+ real NVIDIA requests driven through the proxy — zero leaked 429s, zero rejections.

Wave 1 (40 concurrent) fired instantly. Wave 2 (60 more, fired while wave 1 was still in flight) queued and streamed in across the next window — every single request completed, and not one rate-limit error reached a client.

How It Works

Coding agent ──POST /v1/chat/completions──▶ UnlimitedNIM ──▶ NVIDIA API (40 RPM cap)
                      ▲                          │
                      └──── 200 + SSE stream ◄────┘
                        (or waits in priority queue)

1. A request arrives with an X-Priority header (1/3/5).
2. If the current 60s window has room, it fires immediately.
3. Otherwise it enters the min-heap priority queue — highest priority (lowest number) leaves first; same priority is FIFO.
4. As the window rolls over (or requests finish), the queue drains up to 40/min, never exceeding the limit.
5. Streaming responses relay byte-for-byte; client disconnects abort the upstream call and free the slot (the window slot is never refunded — NVIDIA already counted it, so we never re-fire).

Quick Start

# 1. Set your NVIDIA key
#    (either export it or copy .env.example → .env and fill it in)
export NVIDIA_API_KEY=your-key-here

# 2. Install
pip install -r requirements.txt

# 3. Run
python main.py
# INFO: Uvicorn running on http://0.0.0.0:8000

# 4. Point your agent at http://localhost:8000/v1

Verify it's alive:

curl http://localhost:8000/status

Configuration

All options come from environment variables or a .env file (auto-loaded).

Point Your Agent At It

OpenCode

Add to opencode.json (or ~/.config/opencode/opencode.json):

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "nvidia": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "UnlimitedNIM",
      "options": {
        "baseURL": "http://localhost:8000/v1",
        "apiKey": "anything-works"
      },
      "models": {
        "nvidia/nemotron-3.5-lightning-30b-a3b": {
          "name": "Nemotron 3.5 Lightning 30B"
        }
      }
    }
  }
}

Cline

1. Settings → API Provider → OpenAI Compatible
2. Base URL: http://localhost:8000/v1
3. API Key: anything (the proxy uses its own NVIDIA key)
4. Model: nvidia/nemotron-3.5-lightning-30b-a3b

Roo Code

1. Settings → API Configuration → Provider → OpenAI Compatible
2. Base URL: http://localhost:8000/v1
3. API Key: anything
4. Model: nvidia/nemotron-3.5-lightning-30b-a3b

Load Testing

load_test.py simulates heavy agentic traffic (100 concurrent requests, mixed priorities, two waves) and reports status counts, leaked 429s, and queue wait times:

python load_test.py                       # full run (40 + 60 waves)
python load_test.py --total 12 --wave1 5 --wave2 7   # quick smoke

For safe local runs without touching NVIDIA, start the included mock upstream and point the proxy at it:

python mock_upstream.py                    # terminal 1
NVIDIA_BASE_URL=http://localhost:9001/v1 python main.py   # terminal 2

Project Layout

app/
  config.py        # env-driven settings
  rate_limiter.py  # sliding-window limiter + priority queue + in-flight tracking
  proxy.py         # FastAPI app, streaming upstream, 429 retry, validation
main.py            # entry point
load_test.py       # agentic traffic simulator
mock_upstream.py   # fake NVIDIA upstream for local testing
tests/             # 43 tests covering all 34 test-case specs

Test Coverage

The suite implements the full 34-case spec — window resets, previous-minute in-flight tracking, priority ordering, starvation promotion, clock skew, upstream 429/500 handling, client disconnects, and the compliance layer (/v1/models, /status).

python -m pytest -q   # 43 passed