Ornith-1.0-35B on AMD Strix Halo (Radeon 8060S)

High-performance native deployment package for running Ornith-1.0-35B local multi-agent LLM inference on AMD Ryzen AI Max / Strix Halo (gfx1151) using ROCmFPX and Vulkan.

---

📥 Model Weights on Hugging Face

Download the optimized ROCmFPX GGUF weights directly from Hugging Face: 👉 julianmb/Ornith-1.0-35B-ROCmFPX-StrixHalo on Hugging Face

• Ornith-1.0-35B-ROCmFPX-Speed-StrixHalo.gguf (~19.7 GB, 4.70 bpw, primary production quant)
• Ornith-1.0-35B-ROCmFPX-Quality-StrixHalo.gguf (~32.2 GB, 6.50 bpw, high-precision quant)

---

⚠️ Important: Runtime & Engine Compatibility

[!IMPORTANT] ROCmFPX Tensor Layout Support: ROCmFPX and ROCmFP4 GGUF models utilize custom experimental tensor structures and block layout IDs. Standard upstream llama.cpp, Ollama, LM Studio, and KoboldCPP cannot load ROCmFPX GGUFs out of the box.

How This Package Solves It

To make deployment seamless, this repository includes a pre-compiled, optimized Vulkan engine binary (release/bin/srv-core and release/lib) built specifically for AMD Strix Halo (gfx1151).

• No Compilation Needed: Running python3 run_server.py or ./run.sh automatically executes the bundled ROCmFPX-enabled Vulkan engine.
• Building from Source: If you prefer to compile the engine yourself, build directly from the custom ROCmFPX Fork:

  git clone https://github.com/ciru-ai/ROCmFPX.git
  cd ROCmFPX
  mkdir build && cd build
  cmake -DGGML_VULKAN=ON ..
  cmake --build . --config Release -j$(nproc)
  

---

Key Performance Highlights

• Production Python Launcher: Clean, robust launcher (run_server.py) that sets environment variables, configures memory limits, and handles Vulkan flags automatically.
• 256k Context Window: Full 262,144-token context capacity natively supported.
• 115+ t/s Aggregate Decode: Maximizes 256-bit LPDDR5X memory bandwidth on Strix Halo unified UMA architecture.
• Multi-Agent Concurrency: Supports parallel agent slots with verified 93.8% - 100% tool-calling validity under burst concurrency.
• In-Engine Host Cache: Up to 64 GiB host prompt cache (--cache-ram 65536) delivering instant <0.4s TTFT on multi-turn long prompts.

---

Directory Structure

github/
├── run_server.py              # Production Python server launcher & daemon controller
├── start_bg_server.py         # Persistent background daemon starter script
├── ornith.service             # Systemd user service unit
├── bin/                       # High-performance Vulkan server binaries & libraries
├── LOCAL_VS_REMOTE_PERFORMANCE_COMPARISON.md
├── LICENSE                    # MIT License
└── README.md                  # Project documentation

---

Quick Start

1. Launch Server

Start the production server in the foreground:

python3 run_server.py

Or launch as a persistent background daemon:

python3 start_bg_server.py

2. Verify Endpoint

curl http://127.0.0.1:8089/health
# Output: {"status":"ok"}

---

Advanced Command Line Flags (run_server.py)

Custom Slot Examples:

# High-concurrency 16-agent burst workload (16k context/slot)
python3 run_server.py --slots 16 --ctx-per-slot 16384

# Single prompt 256k ultra-long context session
python3 run_server.py --slots 1 --ctx-per-slot 262144

---

Client Integration Examples

OpenAI Python SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8089/v1",
    api_key="o4a-secret-key"
)

response = client.chat.completions.create(
    model="Ornith-1.0-35B",
    messages=[{"role": "user", "content": "Write a Python script to monitor GPU memory."}]
)
print(response.choices[0].message.content)

Environment Setup (~/.bashrc / Aider / OpenHands / LiteLLM)

export OPENAI_API_KEY="o4a-secret-key"
export OPENAI_BASE_URL="http://127.0.0.1:8089/v1"
export OPENAI_API_BASE="http://127.0.0.1:8089/v1"
export AIDER_MODEL="openai/Ornith-1.0-35B"

---

Recommended Kernel Boot Parameters (AMD Strix Halo)

For maximum Vulkan memory allocation and zero TTM page-eviction throttling, add these parameters to /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="... amdgpu.gttsize=131072 ttm.pages_limit=31457280 amdgpu.cwsr_enable=0 amd_iommu=off"

Update grub and reboot:

sudo update-grub
sudo reboot

---

License

This project is licensed under the MIT License.