djev-run

Serve DiffusionGemma-Jev (djev) on a TypeSafe AI compatible API on Cloud Run with an NVIDIA RTX PRO 6000 Blackwell or NVIDIA L4 GPU, or run directly inside Google Colab Pro (A100 40GB/80GB or L4 24GB) via colab.ipynb, built on mmastrac/djev.

Built-in demo apps:

• /snake: mizorewww/laya-coreml
• /dino: virajbhartiya/laya-vs-jev
• /tetris: trungdq88/jev-tetris

---

Deploy on Google Cloud Run

Follows Cloud Run GPU best practices.

Step 1: Upload Model to GCS

export BUCKET="your-gcs-bucket"
export REGION="us-central1" # Supported RTX PRO 6000 regions: us-central1, europe-west4, asia-southeast1, asia-south2

gcloud storage buckets create "gs://${BUCKET}" --location="${REGION}"
hf download nvidia/diffusiongemma-26B-A4B-it-NVFP4 --local-dir /tmp/dgemma
gcloud storage cp -r /tmp/dgemma/* "gs://${BUCKET}/dgemma/"

Step 2: Deploy to Cloud Run

gcloud beta run deploy djev-dgemma \
  --region="${REGION}" \
  --image=ghcr.io/taeold/djev-run:latest \
  --gpu=1 \
  --gpu-type=nvidia-rtx-pro-6000 \
  --no-gpu-zonal-redundancy \
  --cpu=20 \
  --memory=80Gi \
  --no-cpu-throttling \
  --concurrency=32 \
  --min-instances=0 \
  --max-instances=1 \
  --port=8080 \
  --network=default \
  --subnet=default \
  --vpc-egress=all-traffic \
  --add-volume=name=weights,type=cloud-storage,bucket="${BUCKET}",readonly=false,mount-options=enable-buffered-read=true \
  --add-volume-mount=volume=weights,mount-path=/mnt/gcs \
  --startup-probe=httpGet.path=/health,httpGet.port=8080,initialDelaySeconds=5,periodSeconds=2,timeoutSeconds=2,failureThreshold=120 \
  --set-env-vars="MODEL=/mnt/gcs/dgemma,CANVAS=128,MAX_SEQS=32,MAX_MODEL_LEN=4096,GPU_UTIL=0.40,KV_CACHE_GB=2,ATTN=TRITON_ATTN,TEST_PAGE=1,COPY_TO_SHM=1,ENFORCE_EAGER=1,DISABLE_MM=1,TORCH_COMPILE_DISABLE=1,VLLM_WORKER_MULTIPROC_METHOD=fork,VLLM_UF_EAGER_ALL=1,VLLM_FLASHINFER_MOE_BACKEND=masked_gemm,CUDA_MODULE_LOADING=LAZY"

# --image=ghcr.io/taeold/djev-run:latest: prebuilt from github.com/mmastrac/djev (upstream does not publish a registry image)
# --no-gpu-zonal-redundancy: required for standard regional RTX PRO 6000 quota
# --no-cpu-throttling: keeps all 20 vCPUs active during weight loading and vLLM scheduling
# --network=default --subnet=default --vpc-egress=all-traffic: streams weights from GCS over Google internal networking (~1.05 GiB/s)
# mount-options=enable-buffered-read=true: prefetches 18 GB safetensors shards sequentially from GCS
# TEST_PAGE=1: enables the built-in /snake, /dino, /tetris, and /playground web UIs
# COPY_TO_SHM=1: stages the 17.5 GB model into /dev/shm RAM in the background while Python imports torch/vllm
# VLLM_WORKER_MULTIPROC_METHOD=fork: forks EngineCore from APIServer without re-importing Python
# ENFORCE_EAGER=1 & TORCH_COMPILE_DISABLE=1: skips torch.compile, CUDA graph capture, and redundant startup profiling
# DISABLE_MM=1: skips multimodal encoder initialization for text-only evaluation

Step 3: Open the Built-in Demos

All three demos are standalone HTML files with zero external dependencies that call POST /v1/systemone directly from the browser via fetch():

• Snake: https://<your-cloud-run-url>/snake
• Chrome Dino: https://<your-cloud-run-url>/dino
• Tetris: https://<your-cloud-run-url>/tetris

---

Use with Vercel AI SDK (Optional)

Because djev-run implements the /v1/systemone endpoint contract, you can also point @ai-sdk/typesafe-ai at your Cloud Run URL from Node.js or TypeScript (index.ts):

import { createTypeSafeAi } from '@ai-sdk/typesafe-ai';
import { experimental_evaluate, type Experimental_EvaluationModel } from 'ai';

const typeSafeAi = createTypeSafeAi({
  baseURL: 'https://<your-cloud-run-url>/v1',
});

async function triage(model: Experimental_EvaluationModel, message: string) {
  return experimental_evaluate({
    model,
    state: { message },
    questions: {
      department: {
        type: 'choice',
        instructions: 'Which team should handle this?',
        criteria: {
          billing: 'Payments and refunds',
          support: 'Other requests',
        },
      },
      severity: {
        type: 'score',
        instructions: 'How severe is the issue?',
        criteria: ['Cosmetic', 'Workaround exists', 'Blocking; no workaround'],
      },
      requestsRefund: {
        type: 'boolean',
        instructions: 'Is the customer requesting money back?',
      },
    },
  });
}

const result = await triage(
  typeSafeAi.evaluationModel('jev-latest'),
  'I was charged twice and my account is locked',
);
console.log(result);
// {
//   department: {
//     type: 'choice',
//     choice: 'billing',
//     probabilities: { billing: 0.9988, support: 0.0012 }
//   },
//   severity: {
//     type: 'score',
//     score: 1.9995,
//     probabilities: { '0': 0.0002, '1': 0.0001, '2': 0.9997 }
//   },
//   requestsRefund: {
//     type: 'boolean',
//     probability: 0.9928
//   }
// }

npm install
CLOUD_RUN_URL="https://<your-cloud-run-url>" TYPESAFE_AI_API_KEY="$(gcloud auth print-identity-token)" npm start

---

Performance

• Cold start from zero instances: ~47.5s (down from 4m 05s baseline)
• Median response time: 61 ms on server, 117 ms end-to-end (steps=1, samples=1; 121 ms end-to-end with samples="auto")
• Batch throughput: ~100-123 requests/sec at concurrency=32

JevBench v1.3.0 (N = 231 public suite):

To reach a 47.5-second cold start on Cloud Run, the container streams the 17.5 GB safetensors weights from GCS into /dev/shm RAM in the background (1.05 GiB/s) while Python imports torch and vllm, forks EngineCore from APIServer (VLLM_WORKER_MULTIPROC_METHOD=fork) so modules are not imported twice, disables unused multimodal profiling (DISABLE_MM=1), and skips torch.compile, CUDA graph capture, and redundant memory-profiling passes (ENFORCE_EAGER=1, TORCH_COMPILE_DISABLE=1, --kv-cache-memory).

Empirical benchmarks on Cloud Run with the RTX PRO 6000 confirmed:

• Streaming over VPC egress from GCS FUSE into /dev/shm (~47.5s) outperforms baking weights into the container image (80-115s), avoiding heavy overlay filesystem read overhead and long image import operations.
• --cpu-boost is omitted because host CPU frequency scaling under boosted allocation slows down 20 vCPU container initialization on GPU nodes (73-75s vs 47.5s).

---

Pricing

1 NVIDIA RTX PRO 6000 GPU (20 vCPU, 80 GiB RAM) costs $3.19 per hour while active and scales to $0 when idle with --min-instances=0.