Octolib: Self-Sufficient AI Provider Library

© 2026 Muvon Un Limited (Hong Kong) | Website | Product Page

🚀 Overview

Octolib is a comprehensive, self-sufficient AI provider library that provides a unified, type-safe interface for interacting with multiple AI services. It offers intelligent model selection, robust error handling, and advanced features like cross-provider tool calling and vision support.

✨ Key Features

• 🔌 Multi-Provider Support: OpenAI, Anthropic, xAI, OpenRouter, Cerebras, NVIDIA NIM, Groq, BytePlus, Alibaba Model Studio, Ollama, Together, Featherless, Fireworks, Hetzner, Inception Labs (Mercury), OpenCode Zen/Go, Google Vertex, Google Studio, Amazon, Cloudflare, DeepSeek, MiniMax, Moonshot AI (Kimi), Z.ai, Tinker (Inkling), OctoHub, Local, CLI proxies
• 🛡️ Unified Interface: Consistent API across different providers
• 🔍 Intelligent Model Validation: Strict provider:model format parsing with case-insensitive model support
• 📋 Structured Output: JSON and JSON Schema support for OpenAI, xAI, OpenRouter, DeepSeek, Together, and Z.ai
• 💰 Cost Tracking: Automatic token usage and cost calculation
• 🖼️ Vision Support: Image and video attachment handling for vision-capable models
• 🧰 Tool Calling: Cross-provider tool call standardization
• 🧩 CLI Provider: Use cli:<backend>/<model> (e.g. cli:codex/gpt-5.2-codex). Proxy-only: tools/MCP are not used or controllable.
• ⏱️ Retry Management: Configurable exponential backoff
• 🔒 Secure Design: Environment-based API key management
• ⚙️ Configuration Migration: Reusable, comment-preserving TOML upgrades with locking, versioned backups, and atomic writes
• 🎯 Embedding Support: Multi-provider embedding generation with Jina, Voyage, Google, OpenAI, Together, OctoHub, Local (Ollama, llama.cpp, LM Studio, vLLM), FastEmbed, and HuggingFace
• 🔄 Reranking: Document relevance scoring with cross-encoder models (Voyage AI, Cohere, Jina AI, Mixedbread, Local (llama.cpp, vLLM, TEI), HuggingFace)
• 🎬 Media Generation: Typed image, asynchronous video, speech, and transcription APIs for Cloudflare Workers AI, ElevenLabs, fal, OpenRouter, Replicate, and Runway, with durable jobs and dimensional cost reporting
• ⚖️ Structured Evaluation: Typed yes/no, choice, and score questions answered with calibrated probabilities by TypeSafe's Jev, directly or through Cloudflare AI Gateway

📦 Quick Installation

# From crates.io (recommended) — use the latest version from the badge above
octolib = "<latest>"

# Or latest from git
octolib = { git = "https://github.com/muvon/octolib" }

Every capability is on by default. Pick only what you need to cut compile time and dependencies:

# Chat only — no embedding, reranking, or media stack compiled.
octolib = { version = "<latest>", default-features = false, features = ["llm"] }

octolib::errors, octolib::storage, octolib::utils and set_user_agent are always available. Hardware acceleration is opt-in via features: metal, cuda, cudnn, mkl, accelerate.

🚀 Quick Start

use octolib::{ProviderFactory, ChatCompletionParams, Message};

async fn example() -> anyhow::Result<()> {
    // Parse model and get provider
    let (provider, model) = ProviderFactory::get_provider_for_model("openai:gpt-4o")?;

    // Create messages
    let messages = vec![
        Message::user("Hello, how are you?"),
    ];

    // Create completion parameters
    let params = ChatCompletionParams::new(&messages, &model, 0.7, 1.0, 50, 1000);

    // Get completion (requires OPENAI_API_KEY environment variable)
    let response = provider.chat_completion(params).await?;
    println!("Response: {}", response.content);

    Ok(())
}

Media generation

Media delivery is separate from chat input attachments. The high-level helpers accept the same provider:model addressing used elsewhere and wait for asynchronous jobs when necessary:

use octolib::{generate_image, ImageGenerationRequest};

async fn image_example() -> octolib::MediaResult<()> {
    // Requires OPENROUTER_API_KEY.
    let request = ImageGenerationRequest::new("A red panda astronaut, studio lighting");
    let result = generate_image(
        "openrouter:openai/gpt-image-1",
        request,
    ).await?;

    // OpenRouter reports authoritative request cost in the response.
    let cost_usd = result
        .usage
        .as_ref()
        .and_then(|usage| usage.provider_reported_cost);
    println!("{} image(s), cost: {:?}", result.artifacts.len(), cost_usd);
    Ok(())
}

Six providers are wired, and not every provider serves every task. OpenRouter, Replicate, and fal cover all four task traits: OpenRouter through dedicated endpoints, Replicate through its prediction lifecycle, and fal through its request queue. Runway serves image and video only, against its dated X-Runway-Version contract. ElevenLabs serves speech synthesis and transcription only, and answers synchronously, so its results are complete on submission and there is no job to poll. Cloudflare Workers AI serves image, speech, and transcription the same synchronous way through /ai/run, priced per output tile and step for images and per character or audio minute for audio. Arbitrary model fields live under the provider's own namespace, such as provider_options["replicate"].input or provider_options["fal"].input, with optional field_map mappings for portable fields.

Low-level submit_*, poll_*, and cancel_* methods are public. Persist the credential-free JobHandle to resume work after a restart. A local wait_timeout returns MediaError::WaitTimeout { handle } and does not cancel the remote job. Generated URLs are never downloaded automatically; call download_artifact with an explicit byte limit. That helper accepts HTTPS only, rejects embedded credentials and literal local/private addresses, validates MIME type, and does not follow redirects; applications can impose a stricter DNS/network policy.

Only idempotent schema and polling queries are retried. Generation POSTs are deliberately not replayed after an ambiguous transport failure because doing so can create duplicate paid work.

Cost semantics are strict: provider_reported_cost is used only when the upstream returns a dollar amount. Replicate normally reports compute time rather than dollars, so its cost falls back to a rate — either a caller-supplied CostEstimate or, failing that, this crate's reference table (media::reference_pricing, the media counterpart of the LLM reference_models table, keyed by provider and carrying the model's billing unit). Either way the result is stored as estimated_cost, never disguised as provider-reported cost, and the rate is frozen into the JobHandle at submit so a resumed job prices identically. Every reference rate is an estimate pending verification against the provider's published pricing. See multimodal.md and the media_openrouter / media_replicate examples for the full contract.

Evaluation

Evaluation models answer typed questions about one state with calibrated probabilities instead of generated text. Ask several independent questions in one call and branch on the numbers in code:

use octolib::{evaluate, Answer, EvaluationRequest, Question};

async fn triage() -> octolib::EvaluationResult<()> {
    // Requires TYPESAFE_API_KEY; use "cloudflare:typesafe/jev" to bill AI Gateway credits instead.
    let request = EvaluationRequest::new("Help! My payouts have been failing for 3 days.")
        .with_question("is_urgent", Question::noul("Does this convey urgency?"))
        .with_question(
            "department",
            Question::choice(
                "Which team should handle this?",
                [("billing", "Payments, refunds"), ("technical", "Bugs, outages")],
            ),
        )
        .with_question(
            "frustration",
            Question::score("How frustrated is the customer?", ["Calm", "Frustrated", "Very angry"]),
        );
    let response = evaluate("typesafe:jev-latest", request).await?;
    if let Answer::Noul { noul } = response.answers["is_urgent"] {
        println!("urgent with p={noul:.2}, cost {:?}", response.usage.cost);
    }
    Ok(())
}

Jev bills input tokens only ($0.042 per 1M, output free) and has a 32k context, so trim the state to what the questions need. The response's model field reports the versioned model that answered.

📋 Structured Output

Get structured JSON responses with schema validation:

use octolib::{ProviderFactory, ChatCompletionParams, Message, StructuredOutputRequest};
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize, Debug)]
struct PersonInfo {
    name: String,
    age: u32,
    skills: Vec<String>,
}

async fn structured_example() -> anyhow::Result<()> {
    let (provider, model) = ProviderFactory::get_provider_for_model("openai:gpt-4o")?;

    // Check if provider supports structured output
    if !provider.supports_structured_output(&model) {
        return Err(anyhow::anyhow!("Provider does not support structured output"));
    }

    let messages = vec![
        Message::user("Tell me about a software engineer in JSON format"),
    ];

    // Request structured JSON output
    let structured_request = StructuredOutputRequest::json();
    let params = ChatCompletionParams::new(&messages, &model, 0.7, 1.0, 50, 1000)
        .with_structured_output(structured_request);

    let response = provider.chat_completion(params).await?;

    if let Some(structured) = response.structured_output {
        let person: PersonInfo = serde_json::from_value(structured)?;
        println!("Person: {:?}", person);
    }

    Ok(())
}

🧩 CLI Provider (Proxy Mode)

Use local CLIs as a lightweight proxy. This mode is prompt-only; tool calling/MCP integration is not used or controllable.

let (provider, model) = ProviderFactory::get_provider_for_model("cli:codex/gpt-5.2-codex")?;
// or: "cli:claude/claude-sonnet-4-5"
// or: "cli:gemini/gemini-2.5-pro"
// or: "cli:cursor/auto"

Set a backend-specific command if it is not on PATH:

CLI_CODEX_COMMAND=/path/to/codex
CLI_CLAUDE_COMMAND=/path/to/claude
CLI_GEMINI_COMMAND=/path/to/gemini
CLI_CURSOR_COMMAND=/path/to/cursor-agent

🧰 Tool Calling

Use AI models to call functions with automatic parameter extraction:

use octolib::{ProviderFactory, ChatCompletionParams, Message, FunctionDefinition, ToolCall};
use serde_json::json;

async fn tool_calling_example() -> anyhow::Result<()> {
    let (provider, model) = ProviderFactory::get_provider_for_model("openai:gpt-4o")?;

    // Define available tools/functions
    let tools = vec![
        FunctionDefinition {
            name: "get_weather".to_string(),
            description: "Get the current weather for a location".to_string(),
            parameters: json!({
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }),
            cache_control: None,
        },
        FunctionDefinition {
            name: "calculate".to_string(),
            description: "Perform a mathematical calculation".to_string(),
            parameters: json!({
                "type": "object",
                "properties": {
                    "expression": {
                        "type": "string",
                        "description": "Mathematical expression to evaluate"
                    }
                },
                "required": ["expression"]
            }),
            cache_control: None,
        },
    ];

    let mut messages = vec![
        Message::user("What's the weather in Tokyo and calculate 15 * 23?"),
    ];

    // Initial request with tools
    let params = ChatCompletionParams::new(&messages, &model, 0.7, 1.0, 50, 1000)
        .with_tools(tools.clone());

    let response = provider.chat_completion(params).await?;

    // Check if model wants to call tools
    if let Some(tool_calls) = response.tool_calls {
        println!("Model requested {} tool calls", tool_calls.len());

        // Add assistant's response with tool calls to conversation
        let mut assistant_msg = Message::assistant(&response.content);
        assistant_msg.tool_calls = Some(serde_json::to_value(&tool_calls)?);
        messages.push(assistant_msg);

        // Execute each tool call and add results
        for tool_call in tool_calls {
            println!("Calling tool: {} with args: {}", tool_call.name, tool_call.arguments);

            // Execute the tool (your implementation)
            let result = match tool_call.name.as_str() {
                "get_weather" => {
                    let location = tool_call.arguments["location"].as_str().unwrap_or("Unknown");
                    json!({
                        "location": location,
                        "temperature": 22,
                        "unit": "celsius",
                        "condition": "sunny"
                    })
                }
                "calculate" => {
                    let expr = tool_call.arguments["expression"].as_str().unwrap_or("0");
                    // Simple calculation (in real app, use proper eval)
                    json!({
                        "expression": expr,
                        "result": 345  // 15 * 23
                    })
                }
                _ => json!({"error": "Unknown tool"}),
            };

            // Add tool result to conversation
            messages.push(Message::tool(
                &serde_json::to_string(&result)?,
                &tool_call.id,
                &tool_call.name,
            ));
        }

        // Get final response with tool results
        let params = ChatCompletionParams::new(&messages, &model, 0.7, 1.0, 50, 1000)
            .with_tools(tools);

        let final_response = provider.chat_completion(params).await?;
        println!("Final response: {}", final_response.content);
    } else {
        println!("Direct response: {}", response.content);
    }

    Ok(())
}

Tool Calling Features:

• ✅ Cross-provider support (OpenAI, Anthropic, xAI, Google Vertex, Google Studio, Amazon, OpenRouter, MiniMax, Moonshot, Z.ai, Alibaba, Hetzner, OpenCode, OctoHub, and other OpenAI-compatible providers)
• ✅ Automatic parameter validation via JSON Schema
• ✅ Multi-turn conversations with tool results
• ✅ Parallel tool execution support
• ✅ Standardized ToolCall and GenericToolCall formats across all providers
• ✅ Provider-specific metadata preservation (e.g., Gemini thought signatures)
• ✅ Clean conversion API with to_generic_tool_calls() method

🎯 Embedding Generation

Generate embeddings using multiple providers:

use octolib::embedding::{generate_embeddings, generate_embeddings_batch, InputType};

async fn embedding_example() -> anyhow::Result<()> {
    // Single embedding generation
    let embedding = generate_embeddings(
        "Hello, world!",
        "voyage",  // provider
        "voyage-3.5-lite"  // model
    ).await?;

    println!("Embedding dimension: {}", embedding.len());

    // Batch embedding generation
    let texts = vec![
        "First document".to_string(),
        "Second document".to_string(),
    ];

    let embeddings = generate_embeddings_batch(
        texts,
        "jina",  // provider
        "jina-embeddings-v4",  // model
        InputType::Document,  // input type for better embeddings
        16,  // batch size
        100_000,  // max tokens per batch
    ).await?;

    println!("Generated {} embeddings", embeddings.len());

    Ok(())
}

// Supported embedding providers:
// - Jina: jina-embeddings-v4, jina-clip-v2, etc.
// - Voyage: voyage-3.5, voyage-code-2, etc.
// - Google: gemini-embedding-001, text-embedding-005
// - OpenAI: text-embedding-3-small, text-embedding-3-large
// - FastEmbed: Local models (feature-gated)
// - HuggingFace: sentence-transformers models

🎯 Document Reranking

Improve search results by scoring document relevance with cross-encoder models:

use octolib::reranker::rerank;

async fn reranking_example() -> anyhow::Result<()> {
    let query = "What is machine learning?";
    let documents = vec![
        "Machine learning is a subset of AI.".to_string(),
        "Cooking recipes for beginners.".to_string(),
        "Deep learning uses neural networks.".to_string(),
    ];

    // Rerank documents by relevance to query
    let response = rerank(
        query,
        documents,
        "voyage",           // provider: voyage, cohere, jina, fastembed
        "rerank-2.5",       // model
        Some(2)             // top_k: return top 2 results
    ).await?;

    for (rank, result) in response.results.iter().enumerate() {
        println!("Rank {}: Score {:.4}", rank + 1, result.relevance_score);
        println!("  Document: {}", result.document);
    }

    println!("Total tokens used: {}", response.total_tokens);

    Ok(())
}

// Supported Providers:
//
// API-Based (require API keys):
// - Voyage AI (VOYAGE_API_KEY): rerank-2.5, rerank-2.5-lite, rerank-2, rerank-2-lite
// - Cohere (COHERE_API_KEY): rerank-english-v3.0, rerank-multilingual-v3.0
// - Jina AI (JINA_API_KEY): jina-reranker-v3, jina-reranker-v2-base-multilingual
//
// Local (no API keys, requires features):
// - FastEmbed (fastembed feature): bge-reranker-base, bge-reranker-large, jina-reranker-v1-turbo-en

🔐 OAuth Authentication

Octolib supports OAuth authentication for ChatGPT subscriptions and Anthropic:

OpenAI OAuth (ChatGPT Plus/Pro/Team/Enterprise):

export OPENAI_OAUTH_ACCESS_TOKEN="your_oauth_token"
export OPENAI_OAUTH_ACCOUNT_ID="your_account_id"

Anthropic OAuth:

export ANTHROPIC_OAUTH_TOKEN="your_bearer_token"

The library automatically detects OAuth credentials and prefers them over API keys. See examples/openai_oauth.rs and examples/anthropic_oauth.rs for full usage examples.

🎯 Provider Support Matrix

Structured Output Details

• JSON Mode: Basic JSON object output
• JSON Schema: Full schema validation with strict mode
• Provider Detection: Use provider.supports_structured_output(&model) to check capability

🧠 Thinking/Reasoning Support

Octolib provides first-class support for models that produce thinking/reasoning content. Thinking is stored separately from the main response content, similar to how tool_calls are separate from content.

use octolib::{ProviderFactory, ChatCompletionParams, Message, ThinkingBlock};

async fn thinking_example() -> anyhow::Result<()> {
    // Thinking-capable models: MiniMax, OpenAI o-series, Moonshot (kimi-k2-thinking*), Z.ai, xAI
    let (provider, model) = ProviderFactory::get_provider_for_model("minimax:MiniMax-M2")?;

    let messages = vec![
        Message::user("Solve this complex math problem step by step"),
    ];

    let params = ChatCompletionParams::new(&messages, &model, 0.7, 1.0, 50, 1000);
    let response = provider.chat_completion(params).await?;

    // Access thinking content (separate from response.content)
    if let Some(ref thinking) = response.thinking {
        println!("=== MODEL THINKING ({}) ===", thinking.tokens);
        println!("{}", thinking.content);
        println!("==========================");
    }

    // Final response (clean, no thinking prefix)
    println!("Response: {}", response.content);
    // Token usage breakdown
    if let Some(usage) = &response.exchange.usage {
        println!("Input tokens: {}", usage.input_tokens);
        println!("Cache read tokens: {}", usage.cache_read_tokens);
        println!("Cache write tokens: {}", usage.cache_write_tokens);
        println!("Output tokens: {}", usage.output_tokens);
        println!("Reasoning tokens: {}", usage.reasoning_tokens);
    }
    Ok(())
}

Supported Providers

Token Tracking

Thinking tokens are tracked separately in TokenUsage.reasoning_tokens:

if let Some(usage) = &response.exchange.usage {
    println!("Total tokens: {}", usage.total_tokens);
    println!("  - Input: {}", usage.input_tokens);
    println!("  - Cache Read: {}", usage.cache_read_tokens);
    println!("  - Cache Write: {}", usage.cache_write_tokens);
    println!("  - Output: {}", usage.output_tokens);
    println!("  - Reasoning: {}", usage.reasoning_tokens);
}

📚 Complete Documentation

📖 Quick Navigation

• Overview - Library introduction and core concepts
• Installation Guide - Setup and configuration
• Advanced Usage - Advanced features and customization
• Advanced Guide - Comprehensive usage patterns
• Embedding Guide - Embedding generation with multiple providers
• Reranking Guide - Document relevance scoring
• Tool Calling - Cross-provider tool calling
• Thinking/Reasoning - Reasoning model support
• Configuration Migration - Versioned TOML upgrades and safe file persistence
• Media and Multimodal Providers - Architecture, OpenRouter/Replicate contracts, durable jobs, artifacts, and cost semantics

🌐 Supported Providers

🔒 Privacy & Security

• 🏠 Local-first design
• 🔑 Secure API key management
• 📁 Respects .gitignore
• 🛡️ Comprehensive error handling

🤝 Support & Community

• 🐛 Issues: GitHub Issues
• 📧 Email: opensource@muvon.io
• 🏢 Company: Muvon Un Limited (Hong Kong)

⚖️ License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

---

Built with ❤️ by the Muvon team in Hong Kong