Awesome local LLM

A curated list of awesome platforms, tools, practices and resources that helps run LLMs locally

Table of Contents

• Inference platforms
• Inference engines
• User Interfaces
• Large Language Models
  • Explorers, Benchmarks, Leaderboards
  • Model providers
  • Specific models
    • General purpose
    • Coding
    • Multimodal
    • Image
    • Audio
    • Retrieval-Augmented Generation
    • Safeguards
    • Miscellaneous
• Tools
  • Models
  • Agent Frameworks
  • Model Context Protocol
  • Retrieval-Augmented Generation
  • Coding Agents
  • Computer Use
  • Browser Automation
  • Memory Management
  • Testing, Evaluation and Observability
  • Research
  • Training and Fine-tuning
  • Security and Sandboxing
  • Miscellaneous
• Hardware
• Tutorials
  • Models
  • Prompt Engineering
  • Context Engineering
  • Inference
  • Agents
  • Retrieval-Augmented Generation
  • Miscellaneous
• Communities

Inference platforms

• LM Studio - discover, download and run local LLMs
• unsloth - unified web UI for training and running open models like Qwen, DeepSeek, and Gemma locally
• LocalAI - the free, open-source alternative to OpenAI, Claude and others
• jan - an open source alternative to ChatGPT that runs 100% offline on your computer
• ChatBox - user-friendly desktop client app for AI models/LLMs
• lemonade - a local LLM server with GPU and NPU Acceleration

Back to Table of Contents

Inference engines

• ollama - get up and running with LLMs
• llama.cpp - LLM inference in C/C++
• vllm - a high-throughput and memory-efficient inference and serving engine for LLMs
• exo - run your own AI cluster at home with everyday devices
• BitNet - official inference framework for 1-bit LLMs
• sglang - a fast serving framework for large language models and vision language models
• omlx - LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
• Strata - Qwen3.8-Flash-Next on any consumer hardware
• Nano-vLLM - a lightweight vLLM implementation built from scratch
• TensorRT-LLM - provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs
• koboldcpp - run GGUF models easily with a KoboldAI UI
• dynamo - a datacenter scale distributed inference serving framework
• mlx-lm - generate text and fine-tune large language models on Apple silicon with MLX
• mistral.rs - fast, flexible LLM inference
• flashinfer - kernel library for LLM serving
• LiteRT-LM - Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices
• gpustack - simple, scalable AI model deployment on GPU clusters
• mlx-vlm - a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX
• mini-sglang - a lightweight yet high-performance inference framework for Large Language Models
• executorch - on-device AI across mobile, embedded and edge for PyTorch
• LiteRT - Google's on-device framework for high-performance ML & GenAI deployment on edge platforms, via efficient conversion, runtime, and optimization
• distributed-llama - connect home devices into a powerful cluster to accelerate LLM inference
• ik_llama.cpp - llama.cpp fork with additional SOTA quants and improved performance
• tokenspeed - a speed-of-light LLM inference engine
• sonar - large-scale LLM inference engine based on vLLM
• FastFlowLM - run LLMs on AMD Ryzen™ AI NPUs
• krasis - a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware
• llm-scaler - run LLMs on Intel Arc™ Pro B60 and B70 GPUs
• vllm-gfx906 - vLLM for AMD gfx906 GPUs, e.g. Radeon VII / MI50 / MI60

Back to Table of Contents

User Interfaces

• Open WebUI - User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
• Lobe Chat - an open-source, modern design AI chat framework
• Text generation web UI - LLM UI with advanced features, easy setup, and multiple backend support
• SillyTavern - LLM Frontend for Power Users
• Page Assist - Use your locally running AI models to assist you in your web browsing

Back to Table of Contents

Large Language Models

Explorers, Benchmarks, Leaderboards

• Arena - benchmark & compare the best AI models
• AI Models & API Providers Analysis - understand the AI landscape to choose the best model and provider for your use case
• SWE-rebench - a continuously evolving and decontaminated benchmark for software engineering LLMs
• BullshitBench - measure whether AI models challenge nonsensical prompts instead of confidently answering them
• LLM Explorer - explore list of the open-source LLM models
• Dubesor LLM Benchmark table - small-scale manual performance comparison benchmark
• oobabooga benchmark - a list sorted by size (on disk) for each score
• CyberGym - evaluating AI agents' real-world cybersecurity capabilities at scale
• vakra - a benchmark for evaluating multi-hop, multi-source tool-calling in AI agents
• swe-serve - a benchmark for evaluating multi-hop, multi-source tool-calling in AI agents

Back to Table of Contents

Model providers

• Qwen - powered by Alibaba Cloud
• Mistral AI - a pioneering French artificial intelligence startup
• Tencent - a profile of a Chinese multinational technology conglomerate and holding company
• Unsloth AI - focusing on making AI more accessible to everyone (GGUFs etc.)
• bartowski - providing GGUF versions of popular LLMs
• Beijing Academy of Artificial Intelligence - a private non-profit organization engaged in AI research and development
• Open Thoughts - a team of researchers and engineers curating the best open reasoning datasets

Back to Table of Contents

Specific models

General purpose

• DeepSeek-V4 - a collection of the DeepSeek V4 LLMs
• Qwen3.8 - a collection of the latest generation Qwen LLMs
• NVIDIA Nemotron v3 - a family of open models from NVIDIA with open weights, training data and recipes, delivering leading efficiency and accuracy for building specialized AI agents
• Gemma 4 - a family of open models built by Google DeepMind, that are multimodal, handling text and image input (with audio supported on small models) and generating text output
• Mistral Medium 3.5 - The first flaship models from Mistral AI handling instruction-following, reasoning, and coding in a single set of opened-weights
• gpt-oss - a collection of open-weight models from OpenAI, designed for powerful reasoning, agentic tasks, and versatile developer use cases
• gpt-oss-puzzle-88B - a deployment-optimized large language model developed by NVIDIA, derived from OpenAI's gpt-oss-120b
• Hunyuan - a collection of Tencent's open-source efficient LLMs designed for versatile deployment across diverse computational environments
• Phi-4 - a family of small language, multi-modal and reasoning models from Microsoft
• OpenReasoning-Nemotron - a collection of models from NVIDIA, trained on 5M reasoning traces for math, code and science
• Kimi K2.5 - a collection of open-source, native multimodal agentic models from Moonshot AI that advances practical capabilities in long-horizon coding, coding-driven design, proactive autonomous execution, and swarm-based task orchestration
• GLM-5.3 - a Z.ai's flagship model for long-horizon tasks
• Ling-3.0-flash - a native hybrid reasoning model from inclusionAI, operating with 124B total and 5.1B active parameters
• Granite 4.1 - efficient language models from IBM for multilingual generation, coding, RAG, and AI assistant workflows
• Ornith-1.5 - a collection of open-source models for agentic tasks and coding
• EXAONE-4.5 - LG's First Open-Weight Vision-Language Model for Industrial Intelligence
• Step-3.7-Flash - Mixture-of-Experts vision-language model for developers who need to scale agentic workflows that combine perception, search, and reasoning
• Occamy-1.0 - a compact agentic model purpose-built for real-world co-work
• MiniCPM5 - a collection of SOTA on-device LLMs, small yet powerful

Back to Table of Contents

Coding

• Qwen3-Coder-Next - a collection of Qwen's open-weight language models designed specifically for coding agents and local development
• Devstral 2 - a couple of agentic LLMs for software engineering tasks, excelling at using tools to explore codebases, edit multiple files, and power SWE Agents
• Mellum 2.1 - an assistant model trained by JetBrain
• MiniMax-M3 - a native multimodal model with 1M context
• MiniMax-M2 - a collection of SOTA models for real-world dev & agents
• Laguna-S-2.1 - a 118B total parameter Mixture-of-Experts model with 8B activated parameters per token, designed for agentic coding and long-horizon work
• SWE-FastContext - a family of code-search models from Microsoft powering the Explore subagent for coding agents
• AesCode-32B & AesCode-8B - models from Microsoft that generates information-rich visual artifacts such as slides, posters, and dashboards as HTML/CSS
• MusaCoder-27B - a code model developed by Moore Threads for PyTorch-to-CUDA/MUSA native kernel generation

Back to Table of Contents

Multimodal

• Qwen3-Omni - a collection of the natively end-to-end multilingual omni-modal foundation models from Qwen
• GLM-4.6V - a collection of open source multimodal models with native tool use from Zhipu AI

Back to Table of Contents

Image

• Qwen-Image-2.1 - a unified text-to-image generation and image editing model in the Qwen family
• Qwen3-VL - a collection of the most powerful vision-language models in the Qwen series to date
• GLM-Image - an image generation model
• Ming-Image-0.1-Design - a 6B text-to-image model for UI, infographics, posters, and other text-rich visual designs
• Granite Vision - multimodal models from IBM built for visual document analysis and image understanding
• HunyuanImage - a collection of image generation models from Tencent
• HunyuanVideo - a collection of video generation models from Tencent
• Youtu-Parsing-Omni - a model from Tencent that parses documents, images, charts, geometry, audio and video into structured JSON
• Vidi - a collection of models for multimodal video understanding and creation
• FastVLM - a collection of VLMs with efficient vision encoding from Apple
• MiniCPM-o & MiniCPM-V - multimodal models with leading performance
• LFM2.5-VL - a collection of vision-language models, designed for on-device deployment

Back to Table of Contents

Audio

• whisper-large-v3 - a state-of-the-art model for automatic speech recognition (ASR) and speech translation from OpenAI
• Nemotron Speech - a collection of open, state-of-the-art, production‑ready enterprise speech models from NVIDIA for ASR, TTS, Speaker Diarization and S2SOpenAI
• NVIDIA NemotronLabs VoiceChat 11B - a 11B end-to-end, real-time speech full duplex (FD) model from NVIDIA for conversational AI that jointly performs streaming speech understanding and speech generation
• Qwen3-ASR - a collection of models that support language identification and ASR for 52 languages and dialects
• Qwen3-TTS - a collection of TTS models that cover 10 major languages as well as multiple dialectal voice profiles to meet global application needs
• Granite Speech - a collection of compact and efficient speech-language models from IBM, specifically designed for multilingual automatic speech recognition (ASR) and bidirectional automatic speech translation (AST)
• Voxtral-Small-24B-2507 - an enhancement of Mistral Small 3, incorporating state-of-the-art audio input capabilities while retaining best-in-class text performance
• Voxtral-Mini-4B-Realtime-2602 - a multilingual, realtime speech-transcription model and among the first open-source solutions to achieve accuracy comparable to offline systems with a delay of <500ms
• Voxtral-4B-TTS-2603 - frontier, open-weights text-to-speech model that’s fast, instantly adaptable, and produces lifelike speech for voice agents
• chatterbox - first production-grade open-source TTS model
• VibeVoice - a collection of frontier text-to-speech models from Microsoft
• Kitten TTS - a collection of open-source realistic text-to-speech models designed for lightweight deployment and high-quality voice synthesis
• Streaming Sortformer Diarizer 4spk v2.1 - a streaming version of a novel end-to-end neural model for speaker diarization from NVIDIA

Back to Table of Contents

Retrieval-Augmented Generation

• Nemotron RAG - a set of tools to build retrieval-augmented generation (RAG) systems, improve search and ranking accuracy, and extract structured data from complex docs
• EmbeddingGemma 2 - an open multimodal embedding model built by Google DeepMind which maps text (incl. code), images, video, and audio inputs—and combinations thereof—into a single, unified 768-dimensional vector space
• Qwen3-Embedding - a collection of the latest proprietary Qwen models, specifically designed for text embedding and ranking tasks
• Qwen3-VL-Embedding - an addition to the Qwen embedding models, specifically designed for multimodal information retrieval and cross-modal understanding
• Qwen3-Reranker - a collection of the latest proprietary Qwen models, engineered to refine embedding results
• Qwen3-VL-Reranker - an addition to the Qwen embedding models, specifically designed for multimodal information retrieval and cross-modal understanding

Back to Table of Contents

Safeguards

• Shieldstral 1.0 3B - a compact 3B-parameter, policy-adaptive multimodal safety classifier
• Granite Guardian - a collection of safety models from IBM for detecting risks, toxicity, and hallucinations in LLM workflows
• Qwen3Guard - a collection of safety moderation models built upon Qwen3
• NemoGuard - a collection of models from NVIDIA for content safety, topic-following and security guardrails
• Nemotron-3.5-Content-Safety - a small language model (SLM) that uses Google's Gemma-3-4B-it as the base and is fine-tuned by NVIDIA on multimodal, multilingual, and reasoning-oriented content-safety datasets
• Privasis - a collection of lightweight text-sanitization models from NVIDIA designed to remove or abstract sensitive information from text according to a user-provided sanitization instruction
• SingGuard - a collection of policy-adaptive multimodal LLM Guardrails with dynamic reasoning
• HARC - a family of safety-aligned instruction models from Microsoft trained with HARC
• gpt-oss-safeguard - a collection of safety reasoning models built-upon gpt-oss from OpenAI
• privacy-filter - a bidirectional token-classification model from OpenAI for personally identifiable information (PII) detection and masking in text
• AprielGuard - a safeguard model designed to detect and mitigate both safety risks and security threats in LLM interactions

Back to Table of Contents

Miscellaneous

• Decision 3.0 - a collection of Open multimodal foundation decision models
• GLiNER2.5 - a collection of models for schema-driven information extraction and classification
• clef - a multimodal model from Cloudflare that turns a state and a schema of typed questions into decisions
• d1-3B & d1-omni-600M - open source decision models
• Intern-S2 - a collection of multimodal foundation models for scientific intelligence and long-horizon agents
• Holo4 - a collection of Visual Language Models for computer/mobile use, tool calls, and code
• Marco-MoE - a suit of multilingual MoE models with highly-sparse architectures
• OUI-1 - a diffusion model built for generative UI
• Nemotron-Orchestrator-8B - a state-of-the-art 8B orchestration model designed to solve complex, multi-turn agentic tasks by coordinating a diverse set of expert models and tools
• Arch-Router-1.5B - the fastest LLM router model that aligns to subjective usage preferences
• Waypoint - a collection of real-time interactive video world models
• Hunyuan3D - a collection of everything related (models, datasets etc.) to 3D assets generation from Tencent
• Hunyuan-GameCraft-1.0 - a novel framework for high-dynamic interactive video generation in game environments
• void-model - a model from Netflix that removes objects from videos along with all interactions they induce on the scene — not just secondary effects like shadows and reflections, but physical interactions like objects falling when a person is removed

Back to Table of Contents

Tools

Models

• llmfit - hundreds of models & providers, one command to find what runs on your hardware
• outlines - structured outputs for LLMs
• llama-swap - reliable model swapping for any local OpenAI compatible server - llama.cpp, vllm, etc.
• llguidance - super-fast structured outputs

Back to Table of Contents

Agent Frameworks

• AutoGPT - a powerful platform that allows you to create, deploy, and manage continuous AI agents that automate complex workflows
• langflow - a powerful tool for building and deploying AI-powered agents and workflows
• langchain - build context-aware reasoning applications
• pi - AI agent toolkit: coding agent CLI, unified LLM API, TUI & web UI libraries, Slack bot, vLLM pods
• anything-llm - the all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, MCP compatibility, and more
• autogen - a programming framework for agentic AI
• crewAI - a framework for orchestrating role-playing, autonomous AI agents
• Flowise - build AI agents, visually
• llama_index - the leading framework for building LLM-powered agents over your data
• agno - a full-stack framework for building Multi-Agent Systems with memory, knowledge and reasoning
• sim - open-source platform to build and deploy AI agent workflows
• openai-agents-python - a lightweight, powerful framework for multi-agent workflows
• t3code - an agent harness control surface that enables control of the agents on your machine with a best-in-class mobile app
• NemoClaw - run OpenClaw more securely inside NVIDIA OpenShell with managed inference
• pydantic-ai - a Python agent framework designed to help you quickly, confidently, and painlessly build production grade applications and workflows with Generative AI
• SuperAGI - an open-source framework to build, manage and run useful Autonomous AI Agents
• camel - the first and the best multi-agent framework
• agent-framework - a framework for building, orchestrating and deploying AI agents and multi-agent workflows with support for Python and .NET
• superset - an agentic IDE to orchestrate 100+ coding agents in parallel
• txtai - all-in-one open-source AI framework for semantic search, LLM orchestration and language model workflows
• archgw - a high-performance proxy server that handles the low-level work in building agents: like applying guardrails, routing prompts to the right agent, and unifying access to LLMs, etc.
• genkit - open-source framework for building AI-powered apps in JavaScript, Go, and Python, built and used in production by Google
• emdash - a desktop app for running AI coding agents in parallel
• ClaraVerse - privacy-first, fully local AI workspace with Ollama LLM chat, tool calling, agent builder, Stable Diffusion, and embedded n8n-style automation
• NeMo-Agent-Toolkit - an open-source library for efficiently connecting and optimizing teams of AI agents
• ragbits - building blocks for rapid development of GenAI applications

Back to Table of Contents

Model Context Protocol

• mindsdb - federated query engine for AI - the only MCP Server you'll ever need
• playwright-mcp - Playwright MCP server
• github-mcp-server - GitHub's official MCP Server
• chrome-devtools-mcp - Chrome DevTools for coding agents
• n8n-mcp - a MCP for Claude Desktop / Claude Code / Windsurf / Cursor to build n8n workflows for you
• awslabs/mcp - AWS MCP Servers — helping you get the most out of AWS, wherever you use MCP
• mcp-atlassian - MCP server for Atlassian tools (Confluence, Jira)
• dbhub - zero-dependency, token-efficient database MCP server for Postgres, MySQL, SQL Server, MariaDB, SQLite

Back to Table of Contents

Retrieval-Augmented Generation

• pathway - Python ETL framework for stream processing, real-time analytics, LLM pipelines and RAG
• LightRAG - simple and fast RAG
• graphrag - a modular graph-based RAG system
• onyx - the AI platform connected to your company's docs, apps, and people
• graphiti - build real-time knowledge graphs for AI Agents
• haystack - AI orchestration framework to build customizable, production-ready LLM applications, best suited for building RAG, question answering, semantic search or conversational agent chatbots
• vanna - an open-source Python RAG framework for SQL generation and related functionality
• claude-context - make entire codebase the context for any coding agent
• pipeshub-ai - a fully extensible and explainable workplace AI platform for enterprise search and workflow automation

Back to Table of Contents

Coding Agents

• opencode - a AI coding agent built for the terminal
• zed - a next-generation code editor designed for high-performance collaboration with humans and AI
• OpenHands - a platform for software development agents powered by AI
• cline - autonomous coding agent right in your IDE, capable of creating/editing files, executing commands, using the browser, and more with your permission every step of the way
• goose - an open-source, extensible AI agent that goes beyond code suggestions
• aider - AI pair programming in your terminal
• continue - create, share, and use custom AI code assistants with our open-source IDE extensions and hub of models, rules, prompts, docs, and other building blocks
• tabby - an open-source GitHub Copilot alternative, set up your own LLM-powered code completion server
• void - an open-source Cursor alternative, use AI agents on your codebase, checkpoint and visualize changes, and bring any model or host locally
• crush - the glamourous AI coding agent for your favourite terminal
• kilocode - open source AI coding assistant for planning, building, and fixing code
• Roo-Code - a whole dev team of AI agents in your code editor
• humanlayer - the best way to get AI coding agents to solve hard problems in complex codebases
• openchamber - Agentic Development Environment based on OpenCode AI agent
• 99 - neovim AI agent done right
• ProxyAI - the leading open-source AI copilot for JetBrains

Back to Table of Contents

Computer Use

• open-interpreter - a natural language interface for computers
• cua - the Docker Container for Computer-Use AI Agents
• OmniParser - a simple screen parsing tool towards pure vision based GUI agent
• openwork - an open-source alternative to Claude Cowork, powered by OpenCode
• Agent-S - an open agentic framework that uses computers like a human
• self-operating-computer - a framework to enable multimodal models to operate a computer
• OpenRoom - a browser-based desktop where AI Agent operates every app through natural language, from MiniMaxAI

Back to Table of Contents

Browser Automation

• firecrawl - turn entire websites into LLM-ready markdown or structured data
• browser-use - make websites accessible for AI agents
• puppeteer - a JavaScript API for Chrome and Firefox
• playwright - a framework for Web Testing and Automation
• stagehand - the AI Browser Automation Framework
• nanobrowser - open-source Chrome extension for AI-powered web automation

Back to Table of Contents

Memory Management

• mem0 - universal memory layer for AI Agents
• mempalace - the highest-scoring AI memory system ever benchmarked
• supermemory - memory engine and app that is extremely fast, scalable
• cognee - memory for AI Agents in 5 lines of code
• letta - the stateful agents framework with memory, reasoning, and context management
• LMCache - supercharge your LLM with the fastest KV Cache Layer
• memU - an open-source memory framework for AI companions
• reasoning-bank - a memory mechanism for agents that learns from both successful and failed trajectories, with reasoning stored as memory content

Back to Table of Contents

Testing, Evaluation and Observability

• langfuse - an open-source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with OpenTelemetry, Langchain, OpenAI SDK, LiteLLM, and more
• opik - debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards
• openllmetry - an open-source observability for your LLM application, based on OpenTelemetry
• giskard - an open-source evaluation & testing for AI & LLM systems
• agenta - an open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place
• Evaluator - open-source library for scalable, reproducible evaluation of AI models and benchmarks

Back to Table of Contents

Research

• open-notebook - an open-source implementation of Notebook LM with more flexibility and features
• Perplexica - an open-source alternative to Perplexity AI, the AI-powered search engine
• gpt-researcher - an LLM based autonomous agent that conducts deep local and web research on any topic and generates a long report with citations
• SurfSense - an open-source alternative to NotebookLM / Perplexity / Glean
• RD-Agent - automate the most critical and valuable aspects of the industrial R&D process
• local-deep-researcher - fully local web research and report writing assistant
• local-deep-research - an AI-powered research assistant for deep, iterative research
• maestro - an AI-powered research application designed to streamline complex research tasks

Back to Table of Contents

Training and Fine-tuning

• heretic - fully automatic censorship removal for language models
• sentence-transformers - a Python library for using and training embedding and reranker models for applications like retrieval augmented generation, semantic search, and more
• trl - train transformer language models with reinforcement learning
• OpenRLHF - an easy-to-use, high-performance open-source RLHF framework built on Ray, vLLM, ZeRO-3 and HuggingFace Transformers, designed to make RLHF training simple and accessible
• slime - an LLM post-training framework for RL Scaling
• Kiln - the easiest tool for fine-tuning LLM models, synthetic data generation, and collaborating on datasets
• miles - an enterprise-facing reinforcement learning framework for LLM and VLM post-training, forked from and co-evolving with slime
• OpenEnv - an interface library for RL post training with environments
• RL - scalable toolkit for efficient model reinforcement
• augmentoolkit - train an open-source LLM on new facts
• Gym - evaluate and improve models and agents using environments
• SpecForge - train speculative decoding models effortlessly and port them smoothly to SGLang serving

Back to Table of Contents

Security and Sandboxing

• garak - the LLM vulnerability scanner from NVIDIA
• OpenShell - the safe, private runtime for autonomous AI agents from NVIDIA
• Guardrails - an open-source toolkit from NVIDIA for easily adding programmable guardrails to LLM-based conversational systems
• CubeSandbox - instant, concurrent, secure & lightweight sandbox for AI agents

Back to Table of Contents

Miscellaneous

• context7 - up-to-date code documentation for LLMs and AI code editors
• deepwiki-open - open source DeepWiki: AI-powered wiki generator for GitHub/Gitlab/Bitbucket repositories
• cai - Cybersecurity AI (CAI), the framework for AI Security
• speakr - a personal, self-hosted web application designed for transcribing audio recordings
• presenton - an open-source AI presentation generator and API
• OmniGen2 - exploration to advanced multimodal generation
• 4o-ghibli-at-home - a powerful, self-hosted AI photo stylizer built for performance and privacy
• Observer - local open-source micro-agents that observe, log and react, all while keeping your data private and secure
• mobile-use - a powerful, open-source AI agent that controls your Android or IOS device using natural language
• gabber - build AI applications that can see, hear, and speak using your screens, microphones, and cameras as inputs
• promptcat - a zero-dependency prompt manager/catalog/library in a single HTML file

Back to Table of Contents

Hardware

• Alex Ziskind - tests of pcs, laptops, gpus etc. capable of running LLMs
• Digital Spaceport - reviews of various builds designed for LLM inference
• Donato Capitella - practical and insightful tutorials on running LLMs locally
• JetsonHacks - information about developing on NVIDIA Jetson Development Kits
• Miyconst - tests of various types of hardware capable of running LLMs
• Kolosal - LLM Memory calculator - estimate the RAM requirements of any GGUF model instantly
• LLM Inference VRAM & GPU Requirement Calculator - calculate how many GPUs you need to deploy LLMs
• ZLUDA - CUDA on non-NVIDIA GPUs
• Strix Halo AI Toolboxes - toolboxes for GenAI on AMD Ryzen AI MAX+: containerized environments for LLMs, Image Generation, and Fine-tuning
• Strix Halo Wiki - a website to gather important information and practical guides for systems powered by AMD Ryzen AI MAX and MAX+ processors
• ai-notes - random AI notes for working with local models or playing around with random machine learning bits

Back to Table of Contents

Tutorials

Models

• Let's reproduce GPT-2 (124M)
• nanochat - a full-stack implementation of an LLM like ChatGPT in a single, clean, minimal, hackable, dependency-lite codebase, designed to run on a single 8XH100 node via scripts like speedrun.sh, that run the entire pipeline start to end
• Knowledge Distillation: How LLMs train each other
• gguf-docs - Docs for GGUF quantization (unofficial)
• Embarrassingly Simple Self-Distillation Improves Code Generation

Back to Table of Contents

Prompt Engineering

• Prompt Engineering Guide - guides, papers, lecture, notebooks and resources for prompt engineering
• Prompt Engineering by NirDiamant - a comprehensive collection of tutorials and implementations for Prompt Engineering techniques, ranging from fundamental concepts to advanced strategies
• Prompting guide 101 - a quick-start handbook for effective prompts by Google
• Prompt Engineering by Google - prompt engineering by Google
• Prompt Engineering by Anthropic - prompt engineering by Anthropic
• Prompt Engineering Interactive Tutorial - Prompt Engineering Interactive Tutorial by Anthropic
• Real world prompting - real world prompting tutorial by Anthropic
• Prompt evaluations - prompt evaluations course by Anthropic
• system-prompts-and-models-of-ai-tools - a collection of system prompts extracted from AI tools
• system_prompts_leaks - a collection of extracted System Prompts from popular chatbots like ChatGPT, Claude & Gemini
• Prompt from Codex - Prompt used to steer behavior of OpenAI's Codex

Back to Table of Contents

Context Engineering

• Context-Engineering - a frontier, first-principles handbook inspired by Karpathy and 3Blue1Brown for moving beyond prompt engineering to the wider discipline of context design, orchestration, and optimization
• Awesome-Context-Engineering - a comprehensive survey on Context Engineering: from prompt engineering to production-grade AI systems

Back to Table of Contents

Inference

• vLLM Production Stack - vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization

Back to Table of Contents

Agents

• superpowers - an agentic skills framework & software development methodology that works
• GenAI Agents - tutorials and implementations for various Generative AI Agent techniques
• 500+ AI Agent Projects - a curated collection of AI agent use cases across various industries
• 12-Factor Agents - principles for building reliable LLM applications
• Agents towards production - end-to-end, code-first tutorials covering every layer of production-grade GenAI agents, guiding you from spark to scale with proven patterns and reusable blueprints for real-world launches
• agents.md - a simple, open format for guiding coding agents
• Agent Skills - a simple, open format for giving agents new capabilities and expertise
• skills - Hugging Face Skills are definitions for AI/ML tasks like dataset creation, model training and evaluation
• LLM Agents & Ecosystem Handbook - one-stop handbook for building, deploying, and understanding LLM agents with 60+ skeletons, tutorials, ecosystem guides, and evaluation tools
• 601 real-world gen AI use cases - 601 real-world gen AI use cases from the world's leading organizations by Google
• A practical guide to building agents - a practical guide to building agents by OpenAI

Back to Table of Contents

Retrieval-Augmented Generation

• Pathway AI Pipelines - ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with live data
• RAG Techniques - various advanced techniques for Retrieval-Augmented Generation (RAG) systems
• Controllable RAG Agent - an advanced Retrieval-Augmented Generation (RAG) solution for complex question answering that uses sophisticated graph based algorithm to handle the tasks
• LangChain RAG Cookbook - a collection of modular RAG techniques, implemented in LangChain + Python

Back to Table of Contents

Miscellaneous

• local-llm - everything jamesob knows about running LLMs locally
• Self-hosted AI coding that just works

Back to Table of Contents

Communities

• LocalLLaMA
• LLMDevs
• LocalLLM
• LocalAIServers
• GenAI monitor - monitoring updates & fresh releases related to LLMs, diffusion models and Generative AI

Back to Table of Contents

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines on how to get started.