One click to configure Claude Code to work with local Qwen models
See the codeRun Claude Code with a local LLM on your Mac. Keep your code private.
Qwenvert lets you use Claude Code CLI with a completely local LLM (Qwen2.5-Coder) instead of Anthropic's API. Your code never leaves your machine.
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
│ Claude Code │ --> │ Qwenvert │ --> │ Local Qwen │
│ CLI │ │ (adapter) │ │ Model │
└─────────────┘ └──────────────┘ └─────────────┘
:8088 (via Ollama)
Why? Privacy. Security. Compliance. Zero inference costs. No internet required.
Requirements:
python3 --version)Install from PyPI:
pip install qwenvert
Or install from source:
git clone https://github.com/kmesiab/qwenvert.git
cd qwenvert
pip install -e .
macOS Users (Python 3.11+): If you see an "externally-managed-environment" error, you have two options:
Option 1 (Recommended for development):
git clone https://github.com/kmesiab/qwenvert.git cd qwenvert make venv # Creates .venv virtual environment source .venv/bin/activate make install-dev # Installs qwenvert + dev dependenciesOption 2 (Recommended for end users):
pipx install qwenvert # Installs in isolated environment # Install pipx first if needed: brew install pipxThis is due to PEP 668 which protects system Python on modern macOS.
qwenvert init
This will automatically (no prompts!):
First run takes 2-5 minutes (downloads binaries & models). Subsequent runs are instant.
Example output:
Qwenvert Initialization
✓ Detected: M1 Pro, 16GB RAM, 16 GPU cores, Active cooling
✓ Selected: Qwen2.5 Coder 7B Q5
✓ Downloading from HuggingFace...
✓ Model downloaded: ~/.qwenvert/models/qwen25-coder-7b-q5.gguf (4.2GB)
✓ Configuration saved: ~/.config/qwenvert/config.yaml
Next step: qwenvert start
qwenvert start
You'll see:
Starting Qwenvert
✓ Backend: Ollama with qwen2.5-coder:7b
✓ Backend server: http://localhost:11434 (healthy)
✓ Qwenvert adapter: http://localhost:8088
✓ Ready for Claude Code!
Configure Claude Code:
export ANTHROPIC_BASE_URL=http://localhost:8088
export ANTHROPIC_API_KEY=local-qwen
export ANTHROPIC_MODEL=qwenvert-default
Leave this terminal running.
Missing Dependencies? If Ollama isn't installed, qwenvert will offer to install it automatically:
qwenvert start
You'll see:
======================================================================
Missing Dependency: Ollama
======================================================================
Ollama is not installed (required for running local models)
To install Ollama using Homebrew:
1. Run: brew install ollama
2. Wait for installation to complete
3. Run: qwenvert init
Learn more: https://ollama.ai
======================================================================
Would you like to install Ollama automatically using Homebrew? [Y/n]:
Non-interactive mode:
qwenvert start --auto-install
Automatically installs missing dependencies via Homebrew without prompting.
Note: Auto-installation only works for supported dependencies (Ollama, llama.cpp) when Homebrew is available.
export ANTHROPIC_BASE_URL=http://localhost:8088
export ANTHROPIC_API_KEY=local-qwen
export ANTHROPIC_MODEL=qwenvert-default
claude
That's it! Claude Code now uses your local model. Your code stays on your machine.
Without qwenvert (default):
Claude Code → api.anthropic.com → Claude Sonnet/Opus
(internet) (cloud)
💰 Costs money ☁️ Code leaves machine
With qwenvert (configured):
Claude Code → localhost:8088 → Ollama → Qwen Model
(no internet) (local) (your Mac)
💰 Free 🔒 Code stays local
Claude Code doesn't know the difference - it just uses whatever ANTHROPIC_BASE_URL points to!
# Start qwenvert (terminal 1)
qwenvert start
# Use Claude Code (terminal 2)
export ANTHROPIC_BASE_URL=http://localhost:8088
export ANTHROPIC_API_KEY=local-qwen
export ANTHROPIC_MODEL=qwenvert-default
claude
# When done, stop qwenvert
qwenvert stop
Add to your ~/.zshrc or ~/.bashrc:
# Qwenvert - Local Claude Code
export ANTHROPIC_BASE_URL=http://localhost:8088
export ANTHROPIC_API_KEY=local-qwen
export ANTHROPIC_MODEL=qwenvert-default
Then reload: source ~/.zshrc
Now claude will automatically use qwenvert!
After setting environment variables, verify the setup:
# Check environment variables are set
echo $ANTHROPIC_BASE_URL
# Should show: http://localhost:8088
echo $ANTHROPIC_API_KEY
# Should show: local-qwen
echo $ANTHROPIC_MODEL
# Should show: qwenvert-default
# Make sure qwenvert is running
curl http://localhost:8088/health
# Should return: {"status":"healthy","backend":"connected"}
# Test with Claude Code
claude
# In Claude Code, ask: "What model are you?"
# It should respond as Qwen2.5-Coder (though it might say Claude)
How to tell it's working:
qwenvert monitor shows requests appearingIf it's NOT working:
ANTHROPIC_API_KEY=local-qwenANTHROPIC_BASE_URL and qwenvert is runningANTHROPIC_MODEL=qwenvert-defaultqwenvert status
Output:
Qwenvert Status
Configuration
Model: qwen2.5-coder-7b-q5
Backend: ollama
Backend URL: http://localhost:11434
Adapter: http://localhost:8088
Context Length: 32,768 tokens
Server Health:
Backend: ✓ Running
Adapter: ✓ Running
qwenvert monitor
Shows a live dashboard with:
OpenTelemetry Support: The monitor now uses OpenTelemetry-compliant metrics. Enable OTLP export for integration with observability platforms:
# Enable with local OTLP collector (secure)
export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317
qwenvert monitor --enable-otel
See TELEMETRY_SECURITY.md for complete security details.
Press Ctrl+C to exit.
Check llama-server installation:
qwenvert binary info
Output:
┌──────────────┬────────────────────────────────────────┐
│ Property │ Value │
├──────────────┼────────────────────────────────────────┤
│ Path │ ~/.cache/qwenvert/bin/llama-server │
│ Version │ b3600 │
│ Source │ downloaded │
│ Architecture │ arm64 │
│ Valid │ ✓ Yes │
└──────────────┴────────────────────────────────────────┘
List available versions:
qwenvert binary list
Install specific version:
qwenvert binary install --version b3600
Update to latest:
qwenvert binary update
Verify integrity:
qwenvert binary verify
Rollback to backup:
qwenvert binary rollback
qwenvert backends
Shows which backends (MLX, llama.cpp, Ollama) are available on your system and recommends the fastest option.
Example output on Apple Silicon:
Available Backends:
✓ MLX v0.10.0 (recommended - fastest on Apple Silicon)
✓ llama.cpp b3600 (available)
✗ Ollama (not installed)
qwenvert models list
Output:
Available Models
ID Size RAM Context
qwen2.5-coder-7b-q4 4.1GB 8GB 32K
qwen2.5-coder-7b-q5 4.8GB 16GB 32K
qwen2.5-coder-14b-q4 8.5GB 16GB 32K
qwen2.5-coder-14b-q5 10GB 32GB 32K
Remove downloaded model files to free disk space:
# Interactive selection
qwenvert models clean
# Remove specific model
qwenvert models clean --model-id qwen2.5-coder-7b-instruct-q4_k_m.gguf
# Remove all models (with confirmation)
qwenvert models clean --all
# Preview what would be deleted (dry run)
qwenvert models clean --dry-run
Example output:
Model Cleanup
Models disk usage: 12.3 GB
Available disk space: 45.2 GB
Downloaded models:
1. qwen2.5-coder-7b-instruct-q4_k_m.gguf (4.2 GB)
2. qwen2.5-coder-14b-instruct-q5_k_m.gguf (8.1 GB)
3. All models
4. Cancel
Enter number(s) separated by commas: 1
Models to be deleted:
Filename Size
qwen2.5-coder-7b-instruct-q4_k_m.gguf 4.2 GB
Total space to free: 4.2 GB
Delete these models? [y/N]: y
✓ Cleanup complete! Deleted 1 model(s), freed 4.2 GB
qwenvert hardware
Output:
Hardware Information
Chip: M1 Pro
Total Memory: 16GB
GPU Cores: 16
Performance Cores: 8
Cooling: Active (fan)
Recommended: 32K tokens context
Qwenvert requires one of these backends to run:
brew install ollamaWhen you run qwenvert start, it automatically detects missing dependencies and offers to install them via Homebrew. The following dependencies support auto-installation:
| Dependency | Package Name | Installation Command |
|---|---|---|
| Ollama | ollama | brew install ollama |
| llama.cpp | llama.cpp | (Not yet supported for auto-install) |
Security Note: Auto-installation only works for whitelisted dependencies defined in
ALLOWED_AUTO_INSTALL_DEPENDENCIES. This prevents accidental installation of arbitrary packages.
Interactive (default):
qwenvert start
# Prompts: "Would you like to install Ollama automatically using Homebrew? [Y/n]:"
Non-interactive (CI/automation):
qwenvert start --auto-install
# Automatically installs without prompting
Manual installation (traditional):
# Install Ollama manually
brew install ollama
# Then start qwenvert
qwenvert start
To check if dependencies are installed, qwenvert automatically detects them when you run commands. You can also manually check:
which ollama # Check if Ollama is in PATH
ollama --version # Verify Ollama version
Currently, only Ollama and llama.cpp are supported as backends. Other dependencies (like Homebrew itself) require manual installation.
If you need support for additional backends, please open an issue.
# List models
qwenvert models list
# Re-initialize with different model
qwenvert init --model qwen2.5-coder-14b-q5
# Restart
qwenvert stop
qwenvert start
# Initialize with llama.cpp backend
qwenvert init --backend llamacpp
# Start (same command)
qwenvert start
Why llama.cpp?
Why Ollama? (default)
# Longer context = more memory
qwenvert init --context-length 65536 # 64K tokens
# Shorter context = less memory
qwenvert init --context-length 16384 # 16K tokens
Rule of thumb:
Check if qwenvert is running:
curl http://localhost:8088/health
Should return:
{"status": "healthy", "backend": "connected"}
If not running:
qwenvert start
Problem: HuggingFace download interrupted
Solution:
# Try again (downloads resume automatically)
qwenvert init
# Or download manually and place in ~/.qwenvert/models/
Check memory usage:
qwenvert status
Solutions:
Use smaller model:
qwenvert init --model qwen2.5-coder-7b-q4
Reduce context length:
qwenvert init --context-length 16384
Close other apps to free RAM
Expected speeds:
Enable thermal pacing:
Edit ~/.config/qwenvert/config.yaml:
thermal_pacing: true
thermal_threshold: 70 # Celsius
Or re-run init with thermal protection:
qwenvert init --thermal-pacing
Problem: Python 3.13 not supported yet
Solution: Use Python 3.12 or earlier
# Check version
python3 --version
# Install Python 3.12 via Homebrew
brew install python@3.12
# Use it
pip3.12 install -e .
Problem: Variables reset when you close terminal
Solution: Add to shell config
# Open your shell config
nano ~/.zshrc # or ~/.bashrc for bash
# Add these lines
export ANTHROPIC_BASE_URL=http://localhost:8088
export ANTHROPIC_API_KEY=local-qwen
export ANTHROPIC_MODEL=qwenvert-default
# Save and reload
source ~/.zshrc
Problem: pip install fails with error about externally managed environment
macOS Python 3.11+ Context: Apple now protects system Python to prevent breaking macOS tools. This is PEP 668 in action.
Solution 1 - Virtual Environment (Recommended for development):
# Clone the repository
git clone https://github.com/kmesiab/qwenvert.git
cd qwenvert
# Create and activate virtual environment
make venv
source .venv/bin/activate
# Install
make install-dev
Solution 2 - pipx (Recommended for end users):
# Install pipx if needed
brew install pipx
# Install qwenvert in isolated environment
pipx install qwenvert
Solution 3 - Disable protection (NOT recommended):
# This breaks the system protection - avoid unless you know what you're doing
pip install qwenvert --break-system-packages
Why virtual environments?
Everything. Qwenvert is designed for security-conscious developers.
✅ Your code - Never sent to any server
✅ Prompts - Processed only on your Mac
✅ Responses - Generated locally
✅ Model weights - Stored in ~/.qwenvert/models/
127.0.0.1 only (not accessible from network)Perfect for:
| Mac Type | Model | Speed | Memory | Context |
|---|---|---|---|---|
| 8GB M1 (Air) | 7B Q4 | 15-20 t/s | ~4GB | 16K tokens |
| 16GB M1 Pro | 7B Q5 | 25-35 t/s | ~6GB | 32K tokens |
| 32GB M1 Max | 14B Q5 | 20-30 t/s | ~12GB | 64K tokens |
t/s = tokens per second
| Feature | Qwenvert | Claude API |
|---|---|---|
| Speed | 20-35 t/s | 40-60 t/s |
| Latency | ~0ms (local) | 100-300ms (network) |
| Cost | $0/month | $15-300/month |
| Privacy | 100% local | Cloud |
| Offline | ✅ Yes | ❌ No |
| Code quality | Good | Excellent |
Best for: Security/privacy-critical work, cost-sensitive projects, offline development
Not ideal for: Highest code quality, fastest possible responses
Qwenvert is an HTTP adapter that sits between Claude Code CLI and your local LLM:
Claude Code → Qwenvert → Ollama/llama.cpp → Qwen Model
Not just config - It's a full translation layer:
Ollama has basic Anthropic API support, but:
Qwenvert provides:
Qwenvert supports three backends for running local LLMs: MLX (fastest on Apple Silicon), llama.cpp (fast and cross-platform), and Ollama (easiest setup).
| Backend | Throughput | Performance vs Ollama | Best For |
|---|---|---|---|
| MLX | ~230 tok/s | 1.5-2x faster than llama.cpp | Apple Silicon (M1-M5), Python integration |
| llama.cpp | ~150 tok/s | 3-7x faster than Ollama | Production, cross-platform |
| Ollama | 20-40 tok/s | Baseline | Quick testing, simple setup |
Benchmarks from vLLM-MLX (2026) and Comparative Study (2025)
MLX (Apple's ML framework) is purpose-built for Apple Silicon and provides:
MLX is automatically recommended on Apple Silicon if available.
llama.cpp provides direct Metal GPU acceleration for Apple Silicon, while Ollama adds a Go wrapper layer that introduces overhead:
-ngl 99)| Mac Model | RAM | Model Size | MLX Throughput | llama.cpp Throughput | Expected Response Time |
|---|---|---|---|---|---|
| M1 Air | 8GB | 1.5B Q4 | 45-60 tok/s | 30-40 tok/s | <1 second |
| M1 Pro | 16GB | 7B Q4 | 63-70 tok/s | 28-35 tok/s | 1-2 seconds |
| M2 Max | 32GB | 14B Q4 | 48-55 tok/s | 22-30 tok/s | 2-3 seconds |
| M3 Pro/Max | 18GB+ | 7B Q4 | 65-75 tok/s | 28-35 tok/s | 1-2 seconds |
| M4 Max | 48GB+ | 7B Q4 | 525 tok/s | 150 tok/s | <1 second |
| M5 Pro/Max | 24GB+ | 7B Q4 | 800+ tok/s* | 150 tok/s | <1 second |
*M5 performance based on Apple's official benchmarks with MLX as canonical runtime Performance data from vLLM-MLX research, llama.cpp benchmarks, and Apple ML Research
Use MLX (fastest) if:
Use llama.cpp (cross-platform) if:
Use Ollama (easiest) if:
To switch backends:
qwenvert init --backend llamacpp # Use llama.cpp (default, fastest production backend)
qwenvert init --backend ollama # Use Ollama
MLX Backend (Experimental - Not Yet User-Selectable)
The MLX backend infrastructure is implemented but not yet available via CLI. MLX requires router/launcher integration for in-process execution. Once complete, it will provide 1.5-2x faster inference on Apple Silicon (M1-M5) compared to llama.cpp.
Current status:
For production use, stick with llama.cpp or Ollama backends.
┌─────────────────────────────────────────────────────────────┐
│ Claude Code CLI │
└────────────────────────┬────────────────────────────────────┘
│
POST /v1/messages
│
┌────────────────────────▼────────────────────────────────────┐
│ Qwenvert HTTP Adapter │
│ (localhost:8088) │
│ • Validates requests │
│ • Translates Anthropic → Backend format │
│ • Handles streaming (SSE) │
│ • Monitors performance │
└────────────────────────┬────────────────────────────────────┘
│
Backend-specific API
│
┌────────────────────────▼────────────────────────────────────┐
│ Ollama or llama.cpp Server │
│ (localhost:11434 or :8080) │
└────────────────────────┬────────────────────────────────────┘
│
┌──────▼───────┐
│ Qwen Model │
│ (GGUF) │
└──────────────┘
Optimize for your use case:
Set up convenience aliases:
# Add to ~/.zshrc
alias qw-start='qwenvert start'
alias qw-stop='qwenvert stop'
alias qw-status='qwenvert status'
Monitor performance:
qwenvert monitor
Read advanced docs:
qwenvert monitor)qwenvert statusMeasure qwenvert performance on your Mac:
# Start qwenvert
qwenvert start
# Run benchmarks (separate terminal)
make benchmark
What it tests:
Metrics:
Example output:
┌────────────────┬─────────┬──────┬─────────┬────────┬─────────┬────────┐
│ Benchmark │ Backend │ Quant│ Latency │ Tokens │ Speed │ Status │
├────────────────┼─────────┼──────┼─────────┼────────┼─────────┼────────┤
│ prompt_short │ ollama │ Q4_K │ 1234ms │ 5 │ 4.1 t/s │ ✓ │
│ prompt_medium │ ollama │ Q4_K │ 2456ms │ 89 │ 36.2t/s │ ✓ │
└────────────────┴─────────┴──────┴─────────┴────────┴─────────┴────────┘
Summary:
Average latency: 1845ms
Average throughput: 32.4 tokens/sec
Results saved to benchmarks/results/ for tracking over time.
See benchmarks/README.md for details.
We welcome contributions! Areas where help is needed:
See CONTRIBUTING.md for guidelines.
Apache 2.0 License - see LICENSE
Qwenvert is an independent project and is not affiliated with, endorsed by, or supported by Anthropic. Claude Code is a trademark of Anthropic.
This project implements research-backed development practices for AI agent collaboration:
Our AGENTS.md file follows findings from:
"Repository-Level Instructions Enhance AI Assistant Completion and Efficiency" Li et al., 2025. arXiv:2601.20404 https://arxiv.org/abs/2601.20404
Key findings from the research:
How we apply it:
This approach makes qwenvert development more efficient and maintainable when working with AI coding assistants like Claude Code.
Questions? Issues? Feedback?
Open an issue: https://github.com/kmesiab/qwenvert/issues
Built with care for the Mac M1 community 🚀
Python
98.6%
One click to configure Claude Code to work with local Qwen models
See the codeRun Claude Code with a local LLM on your Mac. Keep your code private.
Qwenvert lets you use Claude Code CLI with a completely local LLM (Qwen2.5-Coder) instead of Anthropic's API. Your code never leaves your machine.
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
│ Claude Code │ --> │ Qwenvert │ --> │ Local Qwen │
│ CLI │ │ (adapter) │ │ Model │
└─────────────┘ └──────────────┘ └─────────────┘
:8088 (via Ollama)
Why? Privacy. Security. Compliance. Zero inference costs. No internet required.
Requirements:
python3 --version)Install from PyPI:
pip install qwenvert
Or install from source:
git clone https://github.com/kmesiab/qwenvert.git
cd qwenvert
pip install -e .
macOS Users (Python 3.11+): If you see an "externally-managed-environment" error, you have two options:
Option 1 (Recommended for development):
git clone https://github.com/kmesiab/qwenvert.git cd qwenvert make venv # Creates .venv virtual environment source .venv/bin/activate make install-dev # Installs qwenvert + dev dependenciesOption 2 (Recommended for end users):
pipx install qwenvert # Installs in isolated environment # Install pipx first if needed: brew install pipxThis is due to PEP 668 which protects system Python on modern macOS.
qwenvert init
This will automatically (no prompts!):
First run takes 2-5 minutes (downloads binaries & models). Subsequent runs are instant.
Example output:
Qwenvert Initialization
✓ Detected: M1 Pro, 16GB RAM, 16 GPU cores, Active cooling
✓ Selected: Qwen2.5 Coder 7B Q5
✓ Downloading from HuggingFace...
✓ Model downloaded: ~/.qwenvert/models/qwen25-coder-7b-q5.gguf (4.2GB)
✓ Configuration saved: ~/.config/qwenvert/config.yaml
Next step: qwenvert start
qwenvert start
You'll see:
Starting Qwenvert
✓ Backend: Ollama with qwen2.5-coder:7b
✓ Backend server: http://localhost:11434 (healthy)
✓ Qwenvert adapter: http://localhost:8088
✓ Ready for Claude Code!
Configure Claude Code:
export ANTHROPIC_BASE_URL=http://localhost:8088
export ANTHROPIC_API_KEY=local-qwen
export ANTHROPIC_MODEL=qwenvert-default
Leave this terminal running.
Missing Dependencies? If Ollama isn't installed, qwenvert will offer to install it automatically:
qwenvert start
You'll see:
======================================================================
Missing Dependency: Ollama
======================================================================
Ollama is not installed (required for running local models)
To install Ollama using Homebrew:
1. Run: brew install ollama
2. Wait for installation to complete
3. Run: qwenvert init
Learn more: https://ollama.ai
======================================================================
Would you like to install Ollama automatically using Homebrew? [Y/n]:
Non-interactive mode:
qwenvert start --auto-install
Automatically installs missing dependencies via Homebrew without prompting.
Note: Auto-installation only works for supported dependencies (Ollama, llama.cpp) when Homebrew is available.
export ANTHROPIC_BASE_URL=http://localhost:8088
export ANTHROPIC_API_KEY=local-qwen
export ANTHROPIC_MODEL=qwenvert-default
claude
That's it! Claude Code now uses your local model. Your code stays on your machine.
Without qwenvert (default):
Claude Code → api.anthropic.com → Claude Sonnet/Opus
(internet) (cloud)
💰 Costs money ☁️ Code leaves machine
With qwenvert (configured):
Claude Code → localhost:8088 → Ollama → Qwen Model
(no internet) (local) (your Mac)
💰 Free 🔒 Code stays local
Claude Code doesn't know the difference - it just uses whatever ANTHROPIC_BASE_URL points to!
# Start qwenvert (terminal 1)
qwenvert start
# Use Claude Code (terminal 2)
export ANTHROPIC_BASE_URL=http://localhost:8088
export ANTHROPIC_API_KEY=local-qwen
export ANTHROPIC_MODEL=qwenvert-default
claude
# When done, stop qwenvert
qwenvert stop
Add to your ~/.zshrc or ~/.bashrc:
# Qwenvert - Local Claude Code
export ANTHROPIC_BASE_URL=http://localhost:8088
export ANTHROPIC_API_KEY=local-qwen
export ANTHROPIC_MODEL=qwenvert-default
Then reload: source ~/.zshrc
Now claude will automatically use qwenvert!
After setting environment variables, verify the setup:
# Check environment variables are set
echo $ANTHROPIC_BASE_URL
# Should show: http://localhost:8088
echo $ANTHROPIC_API_KEY
# Should show: local-qwen
echo $ANTHROPIC_MODEL
# Should show: qwenvert-default
# Make sure qwenvert is running
curl http://localhost:8088/health
# Should return: {"status":"healthy","backend":"connected"}
# Test with Claude Code
claude
# In Claude Code, ask: "What model are you?"
# It should respond as Qwen2.5-Coder (though it might say Claude)
How to tell it's working:
qwenvert monitor shows requests appearingIf it's NOT working:
ANTHROPIC_API_KEY=local-qwenANTHROPIC_BASE_URL and qwenvert is runningANTHROPIC_MODEL=qwenvert-defaultqwenvert status
Output:
Qwenvert Status
Configuration
Model: qwen2.5-coder-7b-q5
Backend: ollama
Backend URL: http://localhost:11434
Adapter: http://localhost:8088
Context Length: 32,768 tokens
Server Health:
Backend: ✓ Running
Adapter: ✓ Running
qwenvert monitor
Shows a live dashboard with:
OpenTelemetry Support: The monitor now uses OpenTelemetry-compliant metrics. Enable OTLP export for integration with observability platforms:
# Enable with local OTLP collector (secure)
export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317
qwenvert monitor --enable-otel
See TELEMETRY_SECURITY.md for complete security details.
Press Ctrl+C to exit.
Check llama-server installation:
qwenvert binary info
Output:
┌──────────────┬────────────────────────────────────────┐
│ Property │ Value │
├──────────────┼────────────────────────────────────────┤
│ Path │ ~/.cache/qwenvert/bin/llama-server │
│ Version │ b3600 │
│ Source │ downloaded │
│ Architecture │ arm64 │
│ Valid │ ✓ Yes │
└──────────────┴────────────────────────────────────────┘
List available versions:
qwenvert binary list
Install specific version:
qwenvert binary install --version b3600
Update to latest:
qwenvert binary update
Verify integrity:
qwenvert binary verify
Rollback to backup:
qwenvert binary rollback
qwenvert backends
Shows which backends (MLX, llama.cpp, Ollama) are available on your system and recommends the fastest option.
Example output on Apple Silicon:
Available Backends:
✓ MLX v0.10.0 (recommended - fastest on Apple Silicon)
✓ llama.cpp b3600 (available)
✗ Ollama (not installed)
qwenvert models list
Output:
Available Models
ID Size RAM Context
qwen2.5-coder-7b-q4 4.1GB 8GB 32K
qwen2.5-coder-7b-q5 4.8GB 16GB 32K
qwen2.5-coder-14b-q4 8.5GB 16GB 32K
qwen2.5-coder-14b-q5 10GB 32GB 32K
Remove downloaded model files to free disk space:
# Interactive selection
qwenvert models clean
# Remove specific model
qwenvert models clean --model-id qwen2.5-coder-7b-instruct-q4_k_m.gguf
# Remove all models (with confirmation)
qwenvert models clean --all
# Preview what would be deleted (dry run)
qwenvert models clean --dry-run
Example output:
Model Cleanup
Models disk usage: 12.3 GB
Available disk space: 45.2 GB
Downloaded models:
1. qwen2.5-coder-7b-instruct-q4_k_m.gguf (4.2 GB)
2. qwen2.5-coder-14b-instruct-q5_k_m.gguf (8.1 GB)
3. All models
4. Cancel
Enter number(s) separated by commas: 1
Models to be deleted:
Filename Size
qwen2.5-coder-7b-instruct-q4_k_m.gguf 4.2 GB
Total space to free: 4.2 GB
Delete these models? [y/N]: y
✓ Cleanup complete! Deleted 1 model(s), freed 4.2 GB
qwenvert hardware
Output:
Hardware Information
Chip: M1 Pro
Total Memory: 16GB
GPU Cores: 16
Performance Cores: 8
Cooling: Active (fan)
Recommended: 32K tokens context
Qwenvert requires one of these backends to run:
brew install ollamaWhen you run qwenvert start, it automatically detects missing dependencies and offers to install them via Homebrew. The following dependencies support auto-installation:
| Dependency | Package Name | Installation Command |
|---|---|---|
| Ollama | ollama | brew install ollama |
| llama.cpp | llama.cpp | (Not yet supported for auto-install) |
Security Note: Auto-installation only works for whitelisted dependencies defined in
ALLOWED_AUTO_INSTALL_DEPENDENCIES. This prevents accidental installation of arbitrary packages.
Interactive (default):
qwenvert start
# Prompts: "Would you like to install Ollama automatically using Homebrew? [Y/n]:"
Non-interactive (CI/automation):
qwenvert start --auto-install
# Automatically installs without prompting
Manual installation (traditional):
# Install Ollama manually
brew install ollama
# Then start qwenvert
qwenvert start
To check if dependencies are installed, qwenvert automatically detects them when you run commands. You can also manually check:
which ollama # Check if Ollama is in PATH
ollama --version # Verify Ollama version
Currently, only Ollama and llama.cpp are supported as backends. Other dependencies (like Homebrew itself) require manual installation.
If you need support for additional backends, please open an issue.
# List models
qwenvert models list
# Re-initialize with different model
qwenvert init --model qwen2.5-coder-14b-q5
# Restart
qwenvert stop
qwenvert start
# Initialize with llama.cpp backend
qwenvert init --backend llamacpp
# Start (same command)
qwenvert start
Why llama.cpp?
Why Ollama? (default)
# Longer context = more memory
qwenvert init --context-length 65536 # 64K tokens
# Shorter context = less memory
qwenvert init --context-length 16384 # 16K tokens
Rule of thumb:
Check if qwenvert is running:
curl http://localhost:8088/health
Should return:
{"status": "healthy", "backend": "connected"}
If not running:
qwenvert start
Problem: HuggingFace download interrupted
Solution:
# Try again (downloads resume automatically)
qwenvert init
# Or download manually and place in ~/.qwenvert/models/
Check memory usage:
qwenvert status
Solutions:
Use smaller model:
qwenvert init --model qwen2.5-coder-7b-q4
Reduce context length:
qwenvert init --context-length 16384
Close other apps to free RAM
Expected speeds:
Enable thermal pacing:
Edit ~/.config/qwenvert/config.yaml:
thermal_pacing: true
thermal_threshold: 70 # Celsius
Or re-run init with thermal protection:
qwenvert init --thermal-pacing
Problem: Python 3.13 not supported yet
Solution: Use Python 3.12 or earlier
# Check version
python3 --version
# Install Python 3.12 via Homebrew
brew install python@3.12
# Use it
pip3.12 install -e .
Problem: Variables reset when you close terminal
Solution: Add to shell config
# Open your shell config
nano ~/.zshrc # or ~/.bashrc for bash
# Add these lines
export ANTHROPIC_BASE_URL=http://localhost:8088
export ANTHROPIC_API_KEY=local-qwen
export ANTHROPIC_MODEL=qwenvert-default
# Save and reload
source ~/.zshrc
Problem: pip install fails with error about externally managed environment
macOS Python 3.11+ Context: Apple now protects system Python to prevent breaking macOS tools. This is PEP 668 in action.
Solution 1 - Virtual Environment (Recommended for development):
# Clone the repository
git clone https://github.com/kmesiab/qwenvert.git
cd qwenvert
# Create and activate virtual environment
make venv
source .venv/bin/activate
# Install
make install-dev
Solution 2 - pipx (Recommended for end users):
# Install pipx if needed
brew install pipx
# Install qwenvert in isolated environment
pipx install qwenvert
Solution 3 - Disable protection (NOT recommended):
# This breaks the system protection - avoid unless you know what you're doing
pip install qwenvert --break-system-packages
Why virtual environments?
Everything. Qwenvert is designed for security-conscious developers.
✅ Your code - Never sent to any server
✅ Prompts - Processed only on your Mac
✅ Responses - Generated locally
✅ Model weights - Stored in ~/.qwenvert/models/
127.0.0.1 only (not accessible from network)Perfect for:
| Mac Type | Model | Speed | Memory | Context |
|---|---|---|---|---|
| 8GB M1 (Air) | 7B Q4 | 15-20 t/s | ~4GB | 16K tokens |
| 16GB M1 Pro | 7B Q5 | 25-35 t/s | ~6GB | 32K tokens |
| 32GB M1 Max | 14B Q5 | 20-30 t/s | ~12GB | 64K tokens |
t/s = tokens per second
| Feature | Qwenvert | Claude API |
|---|---|---|
| Speed | 20-35 t/s | 40-60 t/s |
| Latency | ~0ms (local) | 100-300ms (network) |
| Cost | $0/month | $15-300/month |
| Privacy | 100% local | Cloud |
| Offline | ✅ Yes | ❌ No |
| Code quality | Good | Excellent |
Best for: Security/privacy-critical work, cost-sensitive projects, offline development
Not ideal for: Highest code quality, fastest possible responses
Qwenvert is an HTTP adapter that sits between Claude Code CLI and your local LLM:
Claude Code → Qwenvert → Ollama/llama.cpp → Qwen Model
Not just config - It's a full translation layer:
Ollama has basic Anthropic API support, but:
Qwenvert provides:
Qwenvert supports three backends for running local LLMs: MLX (fastest on Apple Silicon), llama.cpp (fast and cross-platform), and Ollama (easiest setup).
| Backend | Throughput | Performance vs Ollama | Best For |
|---|---|---|---|
| MLX | ~230 tok/s | 1.5-2x faster than llama.cpp | Apple Silicon (M1-M5), Python integration |
| llama.cpp | ~150 tok/s | 3-7x faster than Ollama | Production, cross-platform |
| Ollama | 20-40 tok/s | Baseline | Quick testing, simple setup |
Benchmarks from vLLM-MLX (2026) and Comparative Study (2025)
MLX (Apple's ML framework) is purpose-built for Apple Silicon and provides:
MLX is automatically recommended on Apple Silicon if available.
llama.cpp provides direct Metal GPU acceleration for Apple Silicon, while Ollama adds a Go wrapper layer that introduces overhead:
-ngl 99)| Mac Model | RAM | Model Size | MLX Throughput | llama.cpp Throughput | Expected Response Time |
|---|---|---|---|---|---|
| M1 Air | 8GB | 1.5B Q4 | 45-60 tok/s | 30-40 tok/s | <1 second |
| M1 Pro | 16GB | 7B Q4 | 63-70 tok/s | 28-35 tok/s | 1-2 seconds |
| M2 Max | 32GB | 14B Q4 | 48-55 tok/s | 22-30 tok/s | 2-3 seconds |
| M3 Pro/Max | 18GB+ | 7B Q4 | 65-75 tok/s | 28-35 tok/s | 1-2 seconds |
| M4 Max | 48GB+ | 7B Q4 | 525 tok/s | 150 tok/s | <1 second |
| M5 Pro/Max | 24GB+ | 7B Q4 | 800+ tok/s* | 150 tok/s | <1 second |
*M5 performance based on Apple's official benchmarks with MLX as canonical runtime Performance data from vLLM-MLX research, llama.cpp benchmarks, and Apple ML Research
Use MLX (fastest) if:
Use llama.cpp (cross-platform) if:
Use Ollama (easiest) if:
To switch backends:
qwenvert init --backend llamacpp # Use llama.cpp (default, fastest production backend)
qwenvert init --backend ollama # Use Ollama
MLX Backend (Experimental - Not Yet User-Selectable)
The MLX backend infrastructure is implemented but not yet available via CLI. MLX requires router/launcher integration for in-process execution. Once complete, it will provide 1.5-2x faster inference on Apple Silicon (M1-M5) compared to llama.cpp.
Current status:
For production use, stick with llama.cpp or Ollama backends.
┌─────────────────────────────────────────────────────────────┐
│ Claude Code CLI │
└────────────────────────┬────────────────────────────────────┘
│
POST /v1/messages
│
┌────────────────────────▼────────────────────────────────────┐
│ Qwenvert HTTP Adapter │
│ (localhost:8088) │
│ • Validates requests │
│ • Translates Anthropic → Backend format │
│ • Handles streaming (SSE) │
│ • Monitors performance │
└────────────────────────┬────────────────────────────────────┘
│
Backend-specific API
│
┌────────────────────────▼────────────────────────────────────┐
│ Ollama or llama.cpp Server │
│ (localhost:11434 or :8080) │
└────────────────────────┬────────────────────────────────────┘
│
┌──────▼───────┐
│ Qwen Model │
│ (GGUF) │
└──────────────┘
Optimize for your use case:
Set up convenience aliases:
# Add to ~/.zshrc
alias qw-start='qwenvert start'
alias qw-stop='qwenvert stop'
alias qw-status='qwenvert status'
Monitor performance:
qwenvert monitor
Read advanced docs:
qwenvert monitor)qwenvert statusMeasure qwenvert performance on your Mac:
# Start qwenvert
qwenvert start
# Run benchmarks (separate terminal)
make benchmark
What it tests:
Metrics:
Example output:
┌────────────────┬─────────┬──────┬─────────┬────────┬─────────┬────────┐
│ Benchmark │ Backend │ Quant│ Latency │ Tokens │ Speed │ Status │
├────────────────┼─────────┼──────┼─────────┼────────┼─────────┼────────┤
│ prompt_short │ ollama │ Q4_K │ 1234ms │ 5 │ 4.1 t/s │ ✓ │
│ prompt_medium │ ollama │ Q4_K │ 2456ms │ 89 │ 36.2t/s │ ✓ │
└────────────────┴─────────┴──────┴─────────┴────────┴─────────┴────────┘
Summary:
Average latency: 1845ms
Average throughput: 32.4 tokens/sec
Results saved to benchmarks/results/ for tracking over time.
See benchmarks/README.md for details.
We welcome contributions! Areas where help is needed:
See CONTRIBUTING.md for guidelines.
Apache 2.0 License - see LICENSE
Qwenvert is an independent project and is not affiliated with, endorsed by, or supported by Anthropic. Claude Code is a trademark of Anthropic.
This project implements research-backed development practices for AI agent collaboration:
Our AGENTS.md file follows findings from:
"Repository-Level Instructions Enhance AI Assistant Completion and Efficiency" Li et al., 2025. arXiv:2601.20404 https://arxiv.org/abs/2601.20404
Key findings from the research:
How we apply it:
This approach makes qwenvert development more efficient and maintainable when working with AI coding assistants like Claude Code.
Questions? Issues? Feedback?
Open an issue: https://github.com/kmesiab/qwenvert/issues
Built with care for the Mac M1 community 🚀
Python
98.6%