LLM Server Project

Complete LLM server setup for DGX Spark (128GB RAM, Blackwell GB10). All files are organized in this folder.

📁 Folder Structure

llm_server/
├── 🚀 vLLM Servers (RECOMMENDED):
│   ├── vllm_server.py              # vLLM-optimized server (2-3x faster, 50% less RAM)
│   ├── install_vllm.sh             # Install vLLM in isolated environment
│   └── run_vllm.sh                 # Quick launcher for vLLM server
│
├── 📄 Legacy Server (HuggingFace):
│   └── llm_server.py               # Original transformers-based server
│
├── 📖 Documentation:
│   ├── README.md                   # This file
│   └── LLM_SERVER_README.md        # Original usage documentation
│
├── 📥 Download Scripts:
│   ├── download_nvfp4_models.py      # NVFP4 models downloader (cache-aware)
│   ├── download_nvfp4_models.sh      # NVFP4 downloader (hf CLI)
│   ├── download_gguf_models.py       # GGUF quantized models downloader
│   ├── download_gguf_models.sh       # GGUF downloader (hf CLI)
│   ├── download_models.sh            # Quick wrapper for NVFP4 downloads
│   └── download_gguf.sh              # Quick wrapper for GGUF downloads
│
├── 🚀 Run Scripts:
│   ├── run_server.sh                 # Start legacy server (interactive)
│   └── run_headless.sh               # Start legacy server (headless)
│
├── 📝 .gitignore                     # Git ignore rules
│
└── 📊 logs/                          # All download logs
    ├── model_downloads.log
    ├── model_download_failures.log
    ├── gguf_downloads.log
    └── ... (other log files)

🚀 Quick Start

⚡ vLLM Server (RECOMMENDED — 2-3x faster, 50% less memory)

1. Install vLLM (one-time, takes ~15 minutes):

./install_vllm.sh

2. Run vLLM server:

# NVFP4 model (best performance, ~16GB RAM)
./run_vllm.sh nvidia/Gemma-4-31B-IT-NVFP4 8000

# Or standard model (~20GB RAM)
./run_vllm.sh google/gemma-4-E4B-it 8000

If you already have your own vLLM Python environment, point the launcher to it:

VLLM_PYTHON=/path/to/python ./run_vllm.sh nvidia/Gemma-4-31B-IT-NVFP4 8000

You can also control launcher defaults with a local .env file:

VLLM_MODEL=nvidia/Gemma-4-31B-IT-NVFP4
VLLM_PORT=1111
VLLM_MAX_MODEL_LEN=32768
VLLM_GPU_MEMORY_UTILIZATION=0.60
VLLM_ENFORCE_EAGER=true

Then start with:

./run_vllm.sh

CLI arguments still win over .env values.

3. Test it:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": false
}'

🐌 Legacy Server (HuggingFace transformers)

# Interactive mode (model selection menu)
./run_server.sh

# Headless mode with specific model
./run_headless.sh google/gemma-4-E4B-it 1111

📥 Download Models

# Download NVFP4 models (recommended for vLLM)
./download_models.sh

# Download GGUF models
./download_gguf.sh

🌐 API Endpoints

vLLM Server (port 8000 by default)

• GET / - Server info
• GET /v1/models - List available models (OpenAI format)
• GET /api/tags - List available models (Ollama format)
• POST /v1/chat/completions - Chat completions (OpenAI format, supports tools / tool_calls)
• POST /api/chat - Chat completions (Ollama format; messages like [{\"role\":\"user\",\"content\":\"hello\"}])
• POST /v1/completions - Text completions (OpenAI format)
• POST /api/generate - Text completions (Ollama format)

For reasoning-capable models (for example Gemma 4 and Qwen 3), the OpenAI chat stream can emit short visible [thinking] ... status messages as normal content chunks so simple clients such as VS Code can show progress while the model is still reasoning.

Legacy Server (port 1111 by default)

• GET / - Server info
• GET /api/tags - List available models
• POST /api/generate - Generate text (single prompt)
• POST /api/chat and POST /v1/chat/completions - Chat completions (OpenAI mode supports tools / tool_calls response format)
• POST /api/switch - Switch to a different model
• GET /api/tools - List local file tools (readFiles, editFiles)
• POST /api/tools/execute - Execute one tool call

Local tool execution (readFiles, editFiles)

Set tool sandbox root (optional, defaults to server start directory):

export LLM_TOOL_ROOT=/home/sna/ai-projects

List tools:

curl http://localhost:11434/api/tools

Read files:

curl -X POST http://localhost:11434/api/tools/execute \
  -H "Content-Type: application/json" \
  -d '{"tool":"readFiles","arguments":{"paths":["property-crawler/README.md"]}}'

Edit files (find/replace):

curl -X POST http://localhost:11434/api/tools/execute \
  -H "Content-Type: application/json" \
  -d '{"tool":"editFiles","arguments":{"edits":[{"path":"property-crawler/README.md","find":"Quick Start","replace":"Quick Start (Updated)","all":false}]}}'

📋 Available Models

NVFP4 Models (vLLM optimized, best performance)

Standard Models (legacy server)

All models are downloaded to ~/.cache/huggingface/hub and cached locally.

🔧 Dependencies

vLLM Server

• Python 3.10+
• vLLM (installed via install_vllm.sh)
• fastapi, uvicorn, pydantic

Legacy Server

• Python 3.10+
• torch (with CUDA support)
• transformers
• huggingface_hub (for download scripts)

Virtual environment for legacy server: /home/sna/ai-projects/model-gemma-4/myenv/

📊 Memory Usage Comparison

vLLM Server (NVFP4 quantization)

Legacy Server (HuggingFace transformers, fp16)

Total system RAM: 128 GB — all models fit comfortably with vLLM.

🛑 Stop Server

# Find the process
ps aux | grep vllm_server  # or llm_server

# Kill it
kill <PID>

⚠️ DGX Spark / GB10 (sm_121) Troubleshooting

The GB10 GPU has compute capability 12.1 (sm_121) which is newer than most PyTorch/vLLM binary releases (max sm_120). This causes two issues:

Issue 1: Triton kernel compilation fails

Triton bundles ptxas 12.8 which cannot compile for sm_121a. The fix is already applied automatically by run_vllm.sh and vllm_server.py:

export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas   # uses system CUDA 13.0

Issue 2: "no kernel image is available" crashes

If vLLM still crashes with CUDA kernel errors after the Triton fix, install a PyTorch wheel built with native sm_121 kernels:

./fix_pytorch_sm121.sh

This installs a community-built PyTorch from Qanatpharma/pytorch-sm121-gb10.

Issue 3: torch / torchvision CUDA mismatch

If startup fails with: PyTorch and torchvision were compiled with different CUDA major versions install matching packages in the same env:

pip install --force-reinstall --index-url https://download.pytorch.org/whl/cu130 torchvision torchaudio

Full DGX Spark stack

For a battle-tested full stack (PyTorch + Triton + bitsandbytes + vLLM Docker):

• dgx-spark-llm-stack

💡 Tips

• vLLM is 2-3x faster than legacy server — use it when possible
• NVFP4 models use 40-50% less memory with minimal quality loss
• First request is slow — model needs to warm up
• Streaming mode: Set "stream": true for token-by-token output
• Same network required — devices must share Wi-Fi or use Tailscale
• Use Chatbox/Open WebUI on Mac — point to your DGX IP for a nice UI

📝 Logs

All download logs are stored in the logs/ directory:

• model_downloads.log - NVFP4 download progress and results
• model_download_failures.log - Failed downloads with error messages
• gguf_downloads.log - GGUF download progress and results
• llm_runtime_port_<port>.log - live API request/response generation logs for both servers, including startup model info, prompt previews, streamed chunks, and completion summaries