Mistral AVX Optimized Rust Inference Server

A high-performance, CPU-only inference server for Mistral models, built in Rust with AVX optimizations and OpenAI-compatible API endpoints.

Features

• OpenAI-Compatible API: Drop-in replacement for OpenAI API endpoints
• CPU-Only Inference: No GPU required, optimized for CPU deployment
• AVX Optimizations: Leverages Advanced Vector Extensions for faster computation
• Rust Performance: Built with Rust for memory safety and high performance
• Mistral Models: Specialized for Mistral model architecture

Running the Engine

cargo run --release --features intel-mkl-src -- --prompt "write a basic rust program to teach me rust" --quantized

Performance

• Optimized for modern x86-64 CPUs with AVX support
• Multi-threaded inference
• Memory-efficient model loading
• Streaming response support

Requirements

• Modern x86-64 CPU with AVX support
• Rust 1.70+
• 8GB+ RAM (depending on model size)