AI Assistant Framework

An open-source AI Assistant framework with memory, tool integration, automation workflows, and extensible architecture.

AI Assistant Framework is a modular, local-first framework for building intelligent AI assistants. It provides a complete toolkit for multi-agent orchestration, voice interactions, document intelligence, and business analytics — all while maintaining full control over your data and models.

---

Table of Contents

• Features
• Architecture Overview
• Installation
• Quick Start
• Configuration
• Example Usage
• Roadmap
• Contributing
• License

---

Features

Core Capabilities

Built-in Modules

---

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    AI Assistant Framework                    │
├─────────────────────────────────────────────────────────────┤
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │ Voice Module │  │  BI Dashboard │  │  RAG System  │      │
│  │  (Phone AI)  │  │  (Streamlit)  │  │ (Document QA)│      │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘      │
│         │                  │                  │              │
│         └──────────────────┼──────────────────┘              │
│                            ▼                                │
│              ┌─────────────────────────┐                    │
│              │    Multi-Agent Swarm    │                    │
│              │   (Orchestration Core)  │                    │
│              └───────────┬─────────────┘                    │
│                          │                                  │
│         ┌────────────────┼────────────────┐                │
│         ▼                ▼                ▼                │
│    ┌─────────┐     ┌─────────┐     ┌──────────┐           │
│    │  LLM    │     │  Tools  │     │  Memory  │           │
│    │(Ollama, │     │(Functions│    │ (Context)│           │
│    │OpenAI)  │     │         │     │          │           │
│    └─────────┘     └─────────┘     └──────────┘           │
└─────────────────────────────────────────────────────────────┘

Core Components

1. Swarm Core (swarm/): Multi-agent orchestration engine

   • Agent definition and lifecycle management
   • Dynamic agent handoff and routing
   • Function calling with automatic schema generation
   • Streaming and non-streaming execution modes

2. Agent Types: Pre-built agent implementations

   • Router Agent: Intelligent request routing
   • DB Agent: SQL query execution and database interaction
   • Order Agent: Business logic and external service integration

3. Integration Modules:

   • Voice: FastAPI + WebSocket + Twilio for phone AI
   • Dashboard: Streamlit + Claude for BI analytics
   • RAG: LangChain + Qdrant for document intelligence

For detailed architecture documentation, see docs/architecture.md.

---

Installation

Prerequisites

• Python 3.8 or higher
• Ollama (for local LLM support)
• Qdrant (for RAG vector storage)

Step 1: Clone the Repository

git clone https://github.com/jamesou/ai-assistant-framework.git
cd ai-assistant-framework

Step 2: Install from PyPI (Coming Soon)

pip install ai-assistant-framework

Or install with optional dependencies:

# Voice capabilities
pip install ai-assistant-framework[voice]

# Dashboard capabilities
pip install ai-assistant-framework[dashboard]

# RAG capabilities
pip install ai-assistant-framework[rag]

# All features
pip install ai-assistant-framework[all]

Step 2: Install from Source

pip install -e .

Or for development:

pip install -e ".[dev]"

Step 3: Set Up Environment Variables

Create a .env file in the project root:

cp .env.example .env

Edit .env with your configuration:

# LLM Configuration
LLM_MODEL=qwen2.5-coder:7b

# API Keys (required for specific modules)
OPENAI_API_KEY=your_openai_key_here
ANTHROPIC_TOKEN=your_anthropic_key_here
COHERE_TOKEN=your_cohere_key_here
HF_TOKEN=your_huggingface_token_here

# Voice Module (optional)
MAKE_WEBHOOK_URL=your_make_com_webhook_url
PORT=5050

# Database
DATABASE_URL=sqlite:///rss-feed-database.db

Step 4: Initialize Database (Optional)

python load_sql_data.py

---

Quick Start

1. Multi-Agent Swarm Demo

Start the interactive CLI with the multi-agent system:

python run_swarm.py

Example interaction:

Starting Ollama Swarm CLI
User: Show me the latest news from the database
Router Agent: I'll help you query the database.
Router Agent: transfer_to_db_agent()
DB Agent: What specific information would you like to retrieve?
User: Show me all articles from last week
DB Agent: run_sql_select_statement(sql="SELECT * FROM rss_items WHERE published_date >= date('now', '-7 days')")
...

2. Voice AI Assistant

Start the phone assistant server:

python phone_assistant.py

The server will start on http://localhost:5050. Configure your Twilio webhook to point to /incoming-call.

3. BI Dashboard

Launch the Streamlit dashboard:

streamlit run dashboard.py

Access the dashboard at http://localhost:8501.

4. RAG Document Q&A

Load documents and start the Q&A interface:

# First, load your documents
python load_documents.py

# Then start the RAG system
python langchain_rag.py

---

Configuration

Agent Configuration

Agents are defined in agents.py:

from swarm import Agent

custom_agent = Agent(
    name="Custom Agent",
    model="qwen2.5-coder:7b",
    instructions="You are a specialized agent for...",
    functions=[my_custom_function],
    tool_choice="auto",
    parallel_tool_calls=True
)

Swarm Configuration

Configure the Swarm client for different LLM backends:

from openai import OpenAI
from swarm import Swarm

# Local deployment with Ollama
ollama_client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)
swarm = Swarm(client=ollama_client)

# Or use OpenAI
openai_client = OpenAI(api_key="your-key")
swarm = Swarm(client=openai_client)

Model Support

---

Example Usage

Basic Agent Interaction

from swarm import Swarm, Agent

client = Swarm()

agent = Agent(
    name="Assistant",
    instructions="You are a helpful assistant.",
    model="qwen2.5-coder:7b"
)

response = client.run(
    agent=agent,
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.messages[-1]["content"])

Agent with Tools

from swarm import Agent

def get_weather(location: str):
    """Get weather for a location."""
    return f"The weather in {location} is sunny."

agent = Agent(
    name="Weather Agent",
    instructions="Help users with weather information.",
    functions=[get_weather]
)

Multi-Agent Handoff

from swarm import Agent

sales_agent = Agent(name="Sales", instructions="Handle sales inquiries.")
support_agent = Agent(name="Support", instructions="Handle support tickets.")

def transfer_to_sales():
    """Transfer to sales agent."""
    return sales_agent

def transfer_to_support():
    """Transfer to support agent."""
    return support_agent

router = Agent(
    name="Router",
    instructions="Route to appropriate agent.",
    functions=[transfer_to_sales, transfer_to_support]
)

See the examples directory for more detailed examples.

---

Roadmap

Current (v0.1.x) - Beta

• ✅ Multi-agent swarm orchestration
• ✅ Function calling with automatic schema generation
• ✅ Local LLM support via Ollama
• ✅ Voice AI with Twilio integration
• ✅ BI Dashboard with natural language queries
• ✅ RAG document Q&A system
• ✅ Memory system with InMemory and SQLite backends

Short-term (v0.2.x)

• MCP Integration: Model Context Protocol support for standardized tool calling
• Advanced Agent Handoff: Intent-based routing with confidence scoring
• Knowledge Base: Built-in knowledge base management

Medium-term (v0.3.x)

• Workflow Engine: Visual workflow builder for complex automations
• Multi-Agent Collaboration: Agents working together on complex tasks
• Plugin System: Third-party plugin architecture
• Web UI: Built-in web interface for agent management

Long-term (v1.0)

• Enterprise Features: SSO, audit logging, role-based access
• Advanced Analytics: Usage metrics and performance monitoring
• Model Abstraction: Unified interface for multiple LLM providers
• Deployment Tools: Docker, Kubernetes, and cloud deployment guides

See docs/roadmap.md for detailed roadmap information.

---

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines on:

• Reporting bugs
• Requesting features
• Submitting pull requests
• Development setup

Quick Contributions

# Fork and clone
git clone https://github.com/jamesou/ai-assistant-framework.git

# Create a branch
git checkout -b feature/your-feature

# Make changes and commit
git commit -am "Add new feature"

# Push and create PR
git push origin feature/your-feature

---

License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Acknowledgments

• Built with inspiration from OpenAI Swarm
• Powered by LangChain, Ollama, and Qdrant

---

Support

• 📖 Documentation
• 🐛 Issue Tracker
• 💬 Discussions

---

Status: Beta — Actively developed and open for contributions!