✨ Features

• 🚀 Apple Silicon Optimized - Built on MLX framework for M1/M2/M3/M4 chips
• 🔌 Dual API Support - Compatible with both OpenAI and Anthropic APIs
• 🎯 Complete AI Suite - Chat, audio processing, image generation, embeddings
• ⚡ High Performance - Local inference with hardware acceleration
• 🔐 Privacy-First - All processing happens locally on your machine
• 🛠 Drop-in Replacement - Works with existing OpenAI and Anthropic SDKs

🚀 Installation

pip install mlx-omni-server

⚡ Quick Start

1. Start the server:

   mlx-omni-server
   

2. Choose your preferred API:

   OpenAI API (Click to expand)

   from openai import OpenAI
   
   client = OpenAI(
       base_url="http://localhost:10240/v1",
       api_key="not-needed"
   )
   
   response = client.chat.completions.create(
       model="mlx-community/gemma-3-1b-it-4bit-DWQ",
       messages=[{"role": "user", "content": "Hello!"}]
   )
   print(response.choices[0].message.content)
   

   Anthropic API (Click to expand)

   import anthropic
   
   client = anthropic.Anthropic(
       base_url="http://localhost:10240/anthropic",
       api_key="not-needed"
   )
   
   message = client.messages.create(
       model="mlx-community/gemma-3-1b-it-4bit-DWQ",
       max_tokens=1000,
       messages=[{"role": "user", "content": "Hello!"}]
   )
   print(message.content[0].text)
   

🎉 That's it! You're now running AI locally on your Mac.

📋 API Support

OpenAI Compatible Endpoints (/v1/*)

Anthropic Compatible Endpoints (/anthropic/v1/*)

⚙️ Configuration

# Default (port 10240)
mlx-omni-server

# Custom options
mlx-omni-server --port 8000
MLX_OMNI_LOG_LEVEL=debug mlx-omni-server

# View all options
mlx-omni-server --help

🛠 Development

git clone https://github.com/madroidmaq/mlx-omni-server.git
cd mlx-omni-server
uv sync

# Start with hot-reload
uv run uvicorn mlx_omni_server.main:app --reload --host 0.0.0.0 --port 10240

uv run pytest                    # All tests
uv run pytest tests/chat/openai/ # OpenAI tests
uv run pytest tests/chat/anthropic/ # Anthropic tests

uv run black . && uv run isort . # Format code
uv run pre-commit run --all-files # Run hooks

🎯 Key Features

Model Management

• Auto-discovery of MLX models in HuggingFace cache
• On-demand loading and intelligent caching
• Automatic model downloading when needed

Advanced Capabilities

• Function calling with model-specific parsers
• Real-time streaming for both APIs
• JSON schema validation and structured output
• Extended reasoning (thinking mode) for supported models

📚 Documentation

🔍 Troubleshooting

# Check requirements
python --version  # Should be 3.11+
python -c "import mlx; print(mlx.__version__)"

# Pre-download models (if needed)
huggingface-cli download mlx-community/gemma-3-1b-it-4bit-DWQ

# Enable debug logging
MLX_OMNI_LOG_LEVEL=debug mlx-omni-server

🤝 Contributing

Quick contributor setup:

git clone https://github.com/madroidmaq/mlx-omni-server.git
cd mlx-omni-server
uv sync && uv run pytest