A Curated Collection of resources for applied AI engineering (work in progress).
Python
551
151 commits
updated Jul 25, 2026
README
๐ Awesome LLM Resources
A Curated Collection of LLM resources. ๐กโจ
๐ Updated: 22nd of June 2025
'Serverless' Hosting of Private/OS Models
| Platform/Tool | Rel. | Scale Down | OS ๐ | GH | Start | One-Click | Dev Exp. | Free-Tier |
|---|---|---|---|---|---|---|---|---|
| Baseten | 2019 | > 15 min | ๐ด |  | Guide | ๐ก | ๐ | $30 |
| Modal | 2021 | < 1 min | ๐ด |  | Helpers | โ | ๐ | $30/m |
| HF Endpoints | 2023 | > 15 min | ๐ด |  | None Needed | โ | ๐ | โ |
| Replicate | 2019 | < 1 min | ๐ด |  | Guide | ๐ก | ๐คท | โ |
| Sagemaker (Serverless) | 2017 | N/A | ๐ด |  | N/A | โ | โ | 300,000s |
| Lambda w/ EFS (AWS) | 2014 | < 1 min | ๐ด |  | Guide | โ | โ | โ |
| RunPod Serverless | 2022 | > 30s | ๐ด |  | N/A | โ | ๐คท | โ |
| BentoML | 2019 | > 5 min |  |  | Gallery | ๐ก | ๐ | ๐ $10 |
It goes without saying that these platforms can usually do more than LLM serving**
๐งฎ Serverless Compute Pricing & Limits โ Lambda vs Modal (on CPU)
| Platform | ๐ต Compute Unit | ๐ฅ Per-Request Fee | ๐ Free Tier | โฑ๏ธ Max Timeout | ๐ฆ Concurrency Limit |
|---|---|---|---|---|---|
| AWS Lambda + API GW | GB-sec @ $0.000016667 | $0.20/M Lambda + $1.00/M HTTP API calls | 1M req + 400k GB-s/mo (12 mo) + 1M API calls/mo | 15 min | 1,000 per region (can request more) |
| Modal | CPU-s @ $0.0000131 + GiB-s @ $0.00000222 | โ No per-request fee | $30/mo compute credits (Starter) | Func: 24h โฎ HTTP: 150s โ 303 redirect | Starter: 100 containers / 200 req/s โฎ Team: 1,000 containers |
Access Off-the-Shelf OS Models (via API):
| Platform/Tool | Released | GitHub |
|---|---|---|
| Together.ai | N/A | ๐ด |
| Fireworks.ai | N/A |  |
| Replicate | 2019 | |
| Groq | N/A |  |
| DeepInfra | N/A |  |
| Bedrock | N/A |  |
| Lepton | N/A |  |
| Fal.ai | N/A |  |
| VertexAI | N/A |  |
Local Inference
| Framework | Browser Chat ๐ฅ๏ธ | Organization | Open Source | GitHub |
|---|---|---|---|---|
| Llama.cpp | โ | ggerganov |  |  |
| Ollama | โ | Ollama |  |  |
| gpt4all | โ | Nomic.ai |  |  |
| LMStudio | โ | LMStudio AI | ๐ด |  |
| OpenLLM | โ | BentoML |  | |
LLM Serving Frameworks
| Framework | Open Source | GitHub |
|---|---|---|
| vLLM |  |  |
| OpenLLM | | |
| TGI (Text Generation Inference) |  | |
| TensorRT LLM |  |  |
| Ray Serve |  |  |
| LMDeploy |  |  |
| Ollama | | |
| MLC-LLM |  |  |
Building Open-Source LLM Web Chat UIs
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| Text Generation WebUI | oobabooga | A Gradio web UI for Large Language Models. |  |  |
| Jan AI | Jan HQ | An open source alternative to ChatGPT that runs 100% offline on your computer. Multiple engine support (llama.cpp, TensorRT-LLM). |  |  |
| AnythingLLM | Mintplex Labs | The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, and more. |  |  |
| Superagent | Superagent AI | Allows developers to add powerful AI assistants to their applications using LLMs and RAG. |  |  |
| Bionic-GPT | Bionic GPT | A ChatGPT replacement offering generative AI advantages while maintaining strict data confidentiality. |  |  |
| Open WebUI | Open WebUI | A user-friendly web interface for interacting with Large Language Models (LLMs). |  |  |
| Xyne | xynehq | A sleek, minimal web chat interface for interacting with Large Language Models. |  |  |
| Assistant UI | assistant-ui | An open-source ChatGPT-like interface with a clean and responsive design. |  |  |
| Scira | zaidmukaddam | An AI-powered search interface that leverages LLMs for intelligent search results. |  |  |
| Onyx | onyx-dot-app | A customizable and extendable web chat UI for interacting with large language models. |  |  |
| NextChat | ChatGPTNextWeb | A Next.js-based, open-source ChatGPT clone for seamless web interaction. |  |  |
Rent GPUs (Fine-Tuning, Deploying, Training)
| Platform | Templates | Beginner Friendly | GitHub |
|---|---|---|---|
| Brev.dev | Fine-tuning | โ |  |
| Modal | Fine-tuning | โ | |
| Hyperbolic AI | None | โ |  |
| RunPod | None | โ | |
| Paperspace | Fine-tuning | โ |  |
| Colab | Small models only | โ |  |
Fine-Tuning with No-Code UI
| Tool | Beginner Friendly | Open Source | GitHub |
|---|---|---|---|
| Together.ai | โ | โ | N/A |
| Hugging Face AutoTrain | โ | โ |  |
| AutoML | โ | โ |  |
| LLaMA-Factory | โ | โ |  |
| H2O LLM Studio | โ | โ |  |
Fine-Tuning Frameworks
OS Agentic/AI Workflow
| Framework | Open Source | Beginner Friendly | Released | GitHub |
|---|---|---|---|---|
| LangChain |  | โ | 2022 |  |
| LlamaIndex |  | โ | 2023 |  |
| Swarms |  | โ | 2023 |  |
| CrewAI |  | โ | 2023 |  |
| Autogen |  | โ | 2023 |  |
| AutoChain |  | โ | 2023 |  |
| SuperAGI |  | โ | 2023 |  |
| AILegion |  | โ | 2023 |  |
| MemGPT (Letta) |  | โ | 2023 |  |
| uAgents |  | โ | 2023 |  |
| AGiXT |  | โ | 2023 |  |
| Dify |  | โ | 2024 |  |
| TaskingAI |  | โ | 2024 |  |
| Bee Agent Framework |  | โ | 2024 |  |
| Swarms | | โ | 2024 | |
| IoA |  | โ | 2024 |  |
| Upsonic |  | โ | 2024 |  |
| Parlant |  | โ | 2024 |  |
| Rig |  | โ | 2024 |  |
| eliza |  | โ | 2024 |  |
| TensorZero |  | โ | 2024 |  |
| AgentDock |  | โ | 2025 |  |
Top Agentic Frameworks
| Framework | Open Source | Beginner Friendly | Released | GitHub |
|---|---|---|---|---|
| LangGraph |  | โ | 2023 | |
| Flowise |  | โ | 2023 |  |
| Langroid |  | โ | 2023 |  |
| smolagents |  | โ | 2024 | |
| Semantic Kernel |  | โ | 2023 | |
| Atomic Agents |  | โ | 2024 |  |
| Agno |  | โ | 2024 |  |
| PydanticAI | | โ | 2024 | |
| Mastra |  | โ | 2025 |  |
Agentic Frameworks: Core Capabilities
| Framework | Memory & RAG | Multimodality | Multi-agent Support | Observability |
|---|---|---|---|---|
| AgentDock | Built-in RAG system; knowledge base integration | ๐ข Multi-modal (text, voice, tools, APIs) | Visual workflow orchestration & agent chains | Comprehensive LLM traceability & credit tracking |
| Agno | Integrated memory & vector DB/RAG | ๐ข Native (text, image, audio, video) | Supervisor-worker roles | Built-in cloud dashboard/logging |
| LangGraph | Persistent state; easy external integration | ๐ธ Primarily text; extendable via nodes | Hierarchical orchestration | LangSmith integration & graph editor |
| SmolAgents | Built-in short-term; custom long-term | ๐ธ Vision agents via VLMs | Modular multi-agent composition | Minimal; relies on external logging |
| Mastra | Persistent workflows; native RAG pipelines | ๐ข Multi-modal via integrations | Native multi-agent workflows | Built-in OpenTelemetry dashboards |
| Pydantic AI | DI-based memory & RAG integration | ๐ธ Text-first; multimodal via custom DI | Type-safe manual orchestration | Limited; Python logging/OpenTelemetry |
| Atomic Agents | Per-agent memory & RAG (vector DB) | ๐ข Native multi-modal | Explicit chaining of workflows | Minimal; external instrumentation recommended |
| Autogen | Short-term built-in; external long-term | ๐ธ Text-mainly; extensible | Emergent, free-form collaboration | Moderate; internal logging, no dashboard |
| CrewAI | Stateful memory & team-based RAG | ๐ข Diverse modalities (text, image, etc.) | Supervisor-led multi-team workflows | Integrated dashboards for logging & monitoring |
Please see this google sheet with more columns.
Visual AI Agent Builders
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| Rivet | Ironclad | A visual builder to design and deploy AI agent workflows. |  |  |
| PySpur | PySpur-Dev | A tool to build and visualize AI agents seamlessly. |  |  |
| Flowise | FlowiseAI | A noโcode, visual platform for designing AI agent workflows. | | |
๐ฌ Model Call Pricing for agent systems (Text-only (2000 tokens in, 100 token out), Flat Rate)
| Model | ๐ต $ / Call | ๐ฏ 100 Calls | ๐งฎ 1,000 Calls | ๐ 30,000 Calls |
|---|---|---|---|---|
| Gemini Flash 2.0 | $0.00000 | $0.02 | $0.24 | $7.20 |
| GPT-4o mini | $0.00144 | $0.14 | $1.44 | $43.20 |
| GPT-4.1 | $0.00480 | $0.48 | $4.80 | $144.00 |
| Gemini Pro 2.5 | $0.00350 | $0.35 | $3.50 | $105.00 |
| Claude Haiku 3.5 | $0.00200 | $0.20 | $2.00 | $60.00 |
| Claude Sonnet 4 | $0.00750 | $0.75 | $7.50 | $225.00 |
| GPT-4o | $0.01200 | $1.20 | $12.00 | $360.00 |
| OpenAI o3 | $0.02400 | $2.40 | $24.00 | $720.00 |
Agentic Tools (for โbuildingโ)
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| browser-use | browser-use | Integrates browser functionalities into agentic workflows. |  |  |
| code2prompt | mufeedvh | Converts code snippets into actionable prompts for development. |  |  |
| note-gen | codexu | Automatically generates notes and documentation from your code. |  |  |
| refly | refly-ai | Automates code refactoring and prompt generation tasks. |  |  |
| potpie | potpie-ai | A toolkit for prototyping and building AI agent pipelines. |  |  |
| AgentStack | AgentOps-AI | A comprehensive stack for constructing and deploying AI agents. |  |  |
| browser | lightpanda-io | A browserโbased tool designed for integrating agentic functionalities. |  |  |
| Memary | kingjulio8238 | A memory module for retaining context in agent workflows. |  |  |
| open-canvas | langchain-ai | A visual interface for designing agent workflows with LangChain. |  | |
| agent-service-toolkit | JoshuaC215 | A toolkit for building and deploying agent-based services. |  |  |
Virtual Brains
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| Leon | leon-ai | An openโsource personal assistant and automation platform powered by AI. |  |  |
| Khoj | khoj-ai | A virtual brain for organizing and retrieving your knowledge using AI. |  |  |
AI Agents
| Framework | Organization | Open Source | Released | GitHub |
|---|---|---|---|---|
| GPT Engineer | GPT Engineer Org |  | 2023 |  |
| XAgent | OpenBMB |  | 2023 | |
| Bolt.new | StackBlitz |  | 2023 |  |
| Goose | Block |  | 2023 |  |
| AI Hedge Fund | virattt |  | 2023 |  |
| FinRobot | AI4Finance Foundation |  | 2024 |  |
| STORM | Stanford OVAL |  | 2024 |  |
| Multion | MULTI-ON | ๐ด | N/A |  |
| Minion | Minion AI | ๐ด | N/A |  |
Long-Term Memory
| Provider | Community | Founded | GitHub | โญ Stars | Open Source |
|---|---|---|---|---|---|
| Letta | ๐ฌ Active dev community | Oct 2023 |  | 17k | โ Apache-2.0 |
| Zep | ๐ค Moderate community | Aug 2024 |  | 11.6k | โ ๏ธ Graphiti CE (Apache-2.0) |
| MemoRAG | ๐งช Small research group | Sep 2024 |  | 1.8k | โ Apache-2.0 |
| Memary | ๐ง Niche community | April 2024 | | 2.3k | โ MIT |
| Cognee | ๐ Moderate | Aug 2023 |  | 5.8k | โ Apache-2.0 |
| Mem0 | ๐ Fast-growing | June 2023 |  | 35.2k | โ Apache-2.0 |
Memory Features Comparison
| Provider | Based | Optional KG | Self-Editing / Agentic | Rolling Summaries | Categories |
|---|---|---|---|---|---|
| Letta | ๐งฎ Vector | โ ๏ธ Partial | โ Yes | โ ๏ธ Partial (memory blocks) | โ Yes |
| Zep | ๐ง KG | - | โ Yes | โ Auto chat summarization | โ Yes |
| MemoRAG | ๐งฎ Vector | โ No | โ Yes | โ Uses long-range model | โ No |
| Memary | ๐ง KG | - | โ Yes | โ ๏ธ Plans โrewindโ feature | โ Yes |
| Cognee | ๐ง KG | - | โ Yes | โ No auto summaries | โ Yes |
| Mem0 | ๐งฎ Vector | โ No | โ Yes | โ Not explicit | โ Yes |
Enterprise Security (Cloud-Based Use)
| Provider | Enterprise Security |
|---|---|
| Mem0 | ๐ Hosted with encryption, org/project roles, GDPR-friendly delete. Uses Graphlit (SOC 2 not stated). |
| Letta | โ๏ธ Self-hosted or managed server. User auth & ID-partitioned memory. Graphlit-based. No public SSO details. |
| Zep | โ SOC 2 Type 2. Encrypted at rest/in transit, access controls, JWT, and deletion API ("Right to be Forgotten"). |
| MemoRAG | ๐ Self-host |
| Memary | ๐ Self-host |
| Cognee | ๐ Self-host |
Pricing by Monthly Message Volume
| Provider | 1K msgs/mo | 10K msgs/mo | 100K msgs/mo | 1M msgs/mo |
|---|---|---|---|---|
| Mem0 | ๐ Free | ๐ Freeโ$29 | ๐ต $249 | ๐ข Enterprise (custom) |
| Letta | ๐ Free | ๐ต $20 | ๐ต $750 | ๐ข Enterprise (custom) |
| Zep | ๐ Free | ๐ Free | ๐ต ~ $112.50 | ๐ต ~ $1,237 |
| MemoRAG | ๐ป GPU Server (~$150โ300/mo) | ๐ป GPU Server (~$150โ300/mo) | ๐ป Multi-GPU ($500+) | ๐ฅ๏ธ Cluster ($1K+/mo) |
| Self-host | ๐ฅ๏ธ Small VM (~$15/mo) | ๐ฅ๏ธ Small VM (~$15โ20/mo) | ๐ฅ๏ธ Medium VM ($50โ$100/mo) | ๐ฅ๏ธ Large VM ($200+/mo) |
Evaluation Frameworks and add-ons
| Framework | Open Source | Beginner Friendly | Released | GitHub |
|---|---|---|---|---|
| TruLens |  | โ | 2023 |  |
| Promptfoo |  | โ | 2023 |  |
| DeepEval |  | โ | 2024 |  |
| RAGAS |  | โ | 2023 |  |
| OpenAI Evals |  | โ | 2023 |  |
| LangChain OpenEvals |  | โ | 2025 | |
| LangChain AgentEvals |  | โ | 2025 | |
| LlamaIndex Eval | | โ | 2023 | |
Evaluation Frameworks: Core Differences
| Framework | Pytest / CLI Runner | Metrics Ready-made | Synthetic Data Gen | Offline Judge | Model-Agnostic | Safety | Red-Team | Custom Metrics (setup speed) |
|---|---|---|---|---|---|---|---|---|
| DeepEval | ๐ข deepeval test | 40 + | ๐ข deepeval create-dataset | ๐ข | ๐ข | ๐ข | ๐ข | ๐ข G-Eval builder โ minutes (one function) |
| RAGAS | โ (script asserts) | 6 core RAG + ๐ธ | ๐ข KG-based Q-gen | ๐ข | ๐ข | ๐ธ DIY | โ | ๐ข AspectCritic one-liner โ minutes |
| MLflow Evaluate | โ (mlflow.evaluate) | 3-4 | โ BYO | ๐ธ possible | ๐ธ | ๐ข | โ | ๐ข Subclass scorer โ few lines, ~hour |
| OpenAI Evals | ๐ข CLI orchestrator | ~10 templates | ๐ธ helper script | โ | ๐ข | โ | โ | ๐ข Full Python/YAML eval โ flexible but slower |
Co-Pilots
Voice API
| Framework | Open Source | GitHub |
|---|---|---|
| VAPI.ai | ๐ด |  |
| Bland.ai | ๐ด | N/A |
| CallAnnie | ๐ด | N/A |
| RealtimeTTS |  |  |
| RealtimeSTT |  | |
| Coqui TTS |  |  |
Open Source TTS Models
| Model | License | Stars/Likes | Downloads (Last Month) | Repository |
|---|---|---|---|---|
| Kokoro-82M | Apache 2.0 | โญ 3.16k (HF) | ๐ฅ 557,392 | Hugging Face |
| Zonos-v0.1-transformer | Apache 2.0 | โญ 249 (HF) | ๐ฅ 24,240 | Hugging Face |
| XTTS-v2 | Non-Commercial | โค๏ธ 368 (HF) | ๐ฅ 2,545,850 | Hugging Face |
| ChatTTS | AGPL-3.0 | N/A | N/A | GitHub |
| MeloTTS | MIT | N/A | N/A | GitHub |
For more TTS models and rankings, check out the TTS Leaderboard.
LLM Application Frameworks
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| Eino | CloudWeGo | A lightweight LLM application framework for scalable AI solutions. |  |  |
| Conversation Knowledge Mining Solution Accelerator | Microsoft | A solution accelerator for integrating conversation intelligence and knowledge mining using LLMs. |  | |
| Olmocr | AllenAI | An OCR framework optimized for integration with language models. |  |  |
| PDFMathTranslate | Byaidu | A tool for converting and translating mathematical content in PDFs using LLMs. |  |  |
| Podcastfy | souzatharsis | A tool to generate podcasts from written content using LLMs. |  |  |
| Pandas AI | sinaptik-ai | Brings LLM-powered analytics to pandas dataframes. |  |  |
| Ramalama | containers | An LLM application framework for containerized deployment of AI solutions. |  |  |
| Robyn | facebookexperimental | A scalable framework for building LLM applications from Facebook Experimental. |  |  |
| ExtractThinker | enoch3712 | A tool for extracting and synthesizing insights from textual data using LLMs. |  |  |
OS RAG Frameworks
| Framework | Organization | Open Source | Released | GitHub |
|---|---|---|---|---|
| Haystack | deepset.ai |  | 2023 |  |
| RAGflow | Infiniflow |  | 2024 |  |
| txtai | Neuml |  | 2022 |  |
| LLM App | Pathway |  | 2023 |  |
| Cognita | Truefoundry |  | 2024 |  |
| R2R | SciPhi AI |  | 2024 |  |
| Raptor | Parth Sarthi |  | 2024 |  |
| LightRAG | HKUDS |  | 2023 |  |
| PIKE-RAG | Microsoft |  | 2024 | |
| KAG | OpenSPG |  | 2024 |  |
| MemoRAG | qhjqhj00 |  | 2023 | |
See RAG_Techniques if you get stuck (not always needed)
๐ Vector DBs โ FOSS, Performance, Pricing, DevX
| Vector DB | License | โก Perf / Throughput | โฑ๏ธ Latency (Real-World) | โ๏ธ Cloud Pricing / Free Tier | ๐ป Dev Experience |
|---|---|---|---|---|---|
| Qdrant | Apache 2.0 | ๐ฅ Highest RPS, lowest latency in single-node bench (โฅ4ร vs prev. run) | p95 < 10ms for 1M vecs (1 thread) | Always-on 1GB free; pay-go โ $0.014/hr | REST + gRPC; 7 lang clients; filter-aware HNSW; hybrid support; Python โembeddedโ mode |
| Milvus / Zilliz Cloud | Apache 2.0 | ๐ Fastest index build; RPS trails Qdrant for high-dim vecs | p95 โ 10โ20ms @ 1M 768-dim (DiskANN, vendor data) | 5GB free; serverless $0.30/GB-mo; dedicated from $99/mo | New SDK v2 (async, schema cache); Python/Go/Java/Node support |
| Weaviate | BSD-3-Clause | โ๏ธ Least bench gains, but decent recall (95%+) and throughput | โLow-msโ claimed; users report 100โ300ms if misconfigured | Starts $25/mo; 14-day sandbox free | GraphQL + REST; strong SDKs (Py/TS/Go/Java); easy RAG + hybrid templates |
| pgvector | MIT | ๐ฅ 28ร lower p95 & 16ร higher QPS vs Pinecone s1 @ 99% recall (50M) | p95 < 50ms @ 50M 768-dim (Timescale test) | Neon/Supabase offer free Postgres with pgvector (0.5โ1GB, ~200h CPU) | Pure SQL; supports joins + ACID; great for hybrid text + dense queries |
| Redis 8 Vector | AGPLv3 / RSAL / SSPL | ๐งต 3.4ร higher QPS vs Qdrant, 4ร vs Milvus @ โฅ0.98 recall | Sub-ms avg, <10ms under load (vendor); 9.7ร lower than Aurora+pgvector | Redis Cloud: 30MB free, pay-go from $5/mo; Flex $0.007/hr | Redis Vector Library + RAG helpers; OM clients for .NET/Py/JS; fast setup |
๐พ Vector DB Cloud Pricing (2000-char Chunks, ~768-dim)
| # Chunks | Data Size | ๐ฃ Milvus / Zilliz Cloud (Serverless) | ๐ข Qdrant Cloud | ๐ก Weaviate Cloud (โStandardโ) |
|---|---|---|---|---|
| 10k | ~0.07 GB | ๐ Free โ within 5 GB tier | ๐ Free โ fits in 1 GB RAM / 4 GB disk | $25 base + $1.2 dim fee โ $26 |
| 100k | ~0.67 GB | ๐ Still under 5 GB | ๐ Fits with compression in 4 GB disk | $25 + $12.0 dim fee โ $37 |
| 1M | ~6.7 GB | ๐ต โ $2 storage; add vCU fees or $99 dedicated | ๐ต Needs 10 GB cluster โ โ $20/mo | $25 + $120.5 dim fee โ $145 |
| 10M | ~67 GB | ๐ต โ $20 storage; + compute: $100โ150 total | ๐ต Needs 64+ GB โ $120โ150/mo estimate | $25 + $1,204 dim fee โ $1,230 |
๐ง Embedding Cost โ OpenAI (Small Model, per Chunk Size)
| # Chunks | ๐ 1,000 Chars | ๐ 2,000 Chars | ๐ 3,000 Chars |
|---|---|---|---|
| 1,000 | $0.01 | $0.01 | $0.01 |
| 10,000 | $0.05 | $0.10 | $0.15 |
| 100,000 | $0.50 | $1.00 | $1.50 |
| 1,000,000 | $5.00 | $10.00 | $15.00 |
| 10,000,000 | $50.00 | $100.00 | $150.00 |
AI Tools (for โusingโ)
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| magic-resume | JOYCEQL | An AI-powered tool for generating resumes. |  |  |
| VideoCaptioner | WEIFENG2333 | An AI tool for automatically generating video captions. |  |  |
| DeepSeekAI | DeepLifeStudio | Browser extension for invoking the DeepSeek AI large model. |  |  |
| logocreator | Nutlope | A tool for creating logos using AI. |  |  |
| blinkshot | Nutlope | An AI-powered tool for capturing and enhancing screenshots. |  | |
| pollinations | pollinations | A tool for generating creative images and artwork using AI. |  |  |
| PromptWizard | microsoft | A tool to generate, manage, and optimize prompts for AI models. |  | |
| Open-Interface | AmberSahdev | Control Any Computer Using LLMs. |  |  |
| wut | shobrook | LLM for the terminal |  |  |
Training/Optimization
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| transformerlab-app | transformerlab | An application for training and optimizing transformer models. |  |  |
| fluxgym | cocktailpeanut | A gym environment for reinforcement learning training and optimization. |  |  |
| AutoGPTQ | AutoGPTQ | A tool for automating GPT quantization and optimization. |  |  |
AI Models
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| WALDO | stephansturges | An AI model for visual reasoning and object detection. |  |  |
| Janus | deepseek-ai | A multi-modal AI model for advanced data processing. |  |  |
| ModernBERT | AnswerDotAI | A modernized version of BERT for natural language processing tasks. |  |  |
| Magma | microsoft | A scalable AI model for large-scale data analysis. |  | |
| Cosmos-Nemotron | NVlabs | An AI model for advanced image and video processing. |  |  |
| Paints-UNDO | lllyasviel | An interactive AI model for image generation and editing. |  |  |
Monitoring
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| helicone | Helicone | A platform for monitoring and analyzing AI model performance. |  |  |
| langwatch | langwatch | A tool for monitoring outputs and performance of language models. |  |  |
Infrastructure
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| gpustack | gpustack | A toolkit for managing GPU infrastructure for AI workloads. |  |  |
| harbor | av | A repository for containerized AI infrastructure management. |  |  |
Research Papers on Chain-of-Thought Prompting
| Publication Date | Title | ๐ | Authors | Organization | Technique |
|---|---|---|---|---|---|
| January 28, 2022 | Chain-of-Thought Prompting Elicits Reasoning in Large Language Models | ๐ | Jason Wei, et al. | DeepMind | CoT Prompting |
| March 21, 2022 | Self-Consistency Improves Chain of Thought Reasoning in Language Models | ๐ | Xuezhi Wang et al. | DeepMind | CoT with Self-Consistency |
| May 21, 2022 | Least-to-Most Prompting Enables Complex Reasoning in Large Language Models | ๐ | Denny Zhou et al. | DeepMind | Least-to-Most Prompting |
| May 21, 2022 | Large Language Models are Zero-Shot Reasoners | ๐ | Takeshi Kojima, et al. | DeepMind | Zero-shot-CoT |
| October 6, 2022 | ReAct: Synergizing Reasoning and Acting in Language Models | ๐ | Shunyu Yao et al. | Princeton University | ReAct |
| April 1, 2023 | Teaching Large Language Models to Self-Debug | ๐ | Xiang Lisa Li, et al. | DeepMind, Stanford University | Self-Debugging |
| May 6, 2023 | Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models | ๐ | Lei Wang et al. | The Chinese University of Hong Kong, SenseTime Research | Plan-and-Solve Prompting |
| May 23, 2023 | Letโs Verify Step by Step | ๐ | Anya Goyal, et al. | DeepMind | Verification for CoT |
| October 3, 2023 | Large Language Models Cannot Self-Correct Reasoning Yet | ๐ | Qingxiu Dong, et al. | The Chinese University of Hong Kong, Huawei Noah's Ark Lab | Self-Correction in LLMs |
| November 2023 | Universal Self-Consistency for Large Language Model Generation | ๐ | Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, Denny Zhou | DeepMind | Universal Self-Consistency |
| May 17, 2023 | Tree of Thoughts: Deliberate Problem Solving with Large Language Models | ๐ | Shunyu Yao, et al. | Princeton University, DeepMind | Tree-of-Thought |
| February 15, 2024 | Chain-of-Thought Reasoning Without Prompting | ๐ | Xuezhi Wang, Denny Zhou | DeepMind | Chain-of-Thought Decoding |
| March 21, 2024 | ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting | ๐ | Xiaoxue Cheng et al. | Renmin University of China | CoTGenius |
| June 2024 | Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models | ๐ | Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, Yu-Xiong Wang | Language Agent Tree Search (LATS) | |
| May 2024 | Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning | ๐ | Yuxi Xie, et al. | National University of Singapore, DeepMind | MCTS |
| September 18, 2024 | To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning | ๐ | Zayne Sprague, et al. | The University of Texas at Austin, Johns Hopkins University, Princeton University | Meta-analysis of CoT |
| September 25, 2024 | Chain-of-Thoughtlessness? An Analysis of CoT in Planning | ๐ | Kaya Stechly, et al. | Arizona State University | Analysis of CoT in Planning |
| October 18, 2024 | Supervised Chain of Thought | ๐ | Xiang Zhang, Dujian Ding | University of British Columbia | Supervised Chain of Thought |
| October 24, 2024 | On examples: A Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstration | ๐ | Zhiqiang Hu, et al. | Amazon, Michigan State University | Theoretical Analysis of CoT |
CoT Implementations
| Implementation | Link | Author | GitHub Stars | GitHub Followers |
|---|---|---|---|---|
| CoT | chain-of-thought-hub | Franx Yao |  |  |
| CoT | optillm | Codelion |  |  |
| CoT | auto-cot | Amazon Science |  |  |
| CoT | g1 | BKlieger Groq |  |  |
| Decoding CoT | optillm/cot_decoding.py | Codelion | | |
| Tree of Thoughts | tree-of-thought-llm | Princeton NLP |  |  |
| Tree of Thoughts | tree-of-thoughts | Kye Gomez |  | |
| Tree of Thoughts | saplings | Shobrook |  | |
| MCTS | optillm/mcts.py | Codelion | | |
| Graph of Thoughts | graph-of-thoughts | SPCL |  |  |
| Other | CPO | SAIL SG |  |  |
| Other | Everything-of-Thoughts-XoT | Microsoft |  | |
CoT Fine-Tuned Models & Datasets
Models
| Model Name | Author | Size | Link |
|---|---|---|---|
| CoT-T5-3B | KAIST AI | 3B | ๐ |
| CoT-T5-11B | KAIST AI | 11B | ๐ |
| Llama-3.2V-11B-cot | Xkev | 11B | ๐ |
| Llama-3.1-8B-Instruct-Reasoner-1o1_v0.3 | Lyte | 8B | ๐ |
Datasets
| Dataset Name | Author | Data Size | Likes | Link |
|---|---|---|---|---|
| chain-of-thought-sharegpt | Isaiah Bjork | 7.14k rows | ๐ 8 | ๐ |
| CoT-Collection | KAIST AI | 1.84 million rows | ๐ 122 | ๐ |
| Reasoner-1o1-v0.3-HQ | Lyte | 370 rows | ๐ 7 | ๐ |
| OpenLongCoT-Pretrain | qq8933 | 103k rows | ๐ 86 | ๐ |
Learning Resources
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| awesome-cursorrules | PatrickJS | A curated list of resources and guides on cursorrules. |  |  |
| ai-engineering-hub | patchy631 | A hub of AI engineering learning resources, tutorials, and best practices. |  |  |
| GenAI_Agents | NirDiamant | Resources and examples for building Generative AI Agents. |  |  |
| learn-agentic-ai | panaversity | Learning materials for understanding and building agentic AI. |  |  |
| awesome-generative-ai | steven2358 | A curated list of generative AI resources and projects. |  |  |
| awesome-mcp-servers | punkpeye | A curated collection of awesome MCP servers resources. |  |  |
| GenAI-Showcase | mongodb-developer | A showcase of innovative Generative AI projects. |  |  |
| well-architected-iac-analyzer | aws-samples | A tool to analyze and ensure well-architected Infrastructure as Code practices. |  | |
| llama-cookbook | meta-llama | A collection of recipes and guides for working with LLaMA models. |  |  |
| optillm | codelion | Resources for optimizing LLM usage and performance. | | |
| cursor.directory | pontusab | A directory of tools and resources related to cursor-based workflows. |  |  |
| GenAI_Agents | NirDiamant | A curated collection of generative AI agents and related tools. | | |
large-language-models
llm
resources
Contributors
Similar repos
Languages
Python
100.0%
A Curated Collection of resources for applied AI engineering (work in progress).
Python
551
151 commits
updated Jul 25, 2026
README
๐ Awesome LLM Resources
A Curated Collection of LLM resources. ๐กโจ
๐ Updated: 22nd of June 2025
'Serverless' Hosting of Private/OS Models
| Platform/Tool | Rel. | Scale Down | OS ๐ | GH | Start | One-Click | Dev Exp. | Free-Tier |
|---|---|---|---|---|---|---|---|---|
| Baseten | 2019 | > 15 min | ๐ด | | Guide | ๐ก | ๐ | $30 |
| Modal | 2021 | < 1 min | ๐ด | | Helpers | โ | ๐ | $30/m |
| HF Endpoints | 2023 | > 15 min | ๐ด | | None Needed | โ | ๐ | โ |
| Replicate | 2019 | < 1 min | ๐ด | | Guide | ๐ก | ๐คท | โ |
| Sagemaker (Serverless) | 2017 | N/A | ๐ด | | N/A | โ | โ | 300,000s |
| Lambda w/ EFS (AWS) | 2014 | < 1 min | ๐ด | | Guide | โ | โ | โ |
| RunPod Serverless | 2022 | > 30s | ๐ด | | N/A | โ | ๐คท | โ |
| BentoML | 2019 | > 5 min | | | Gallery | ๐ก | ๐ | ๐ $10 |
It goes without saying that these platforms can usually do more than LLM serving**
๐งฎ Serverless Compute Pricing & Limits โ Lambda vs Modal (on CPU)
| Platform | ๐ต Compute Unit | ๐ฅ Per-Request Fee | ๐ Free Tier | โฑ๏ธ Max Timeout | ๐ฆ Concurrency Limit |
|---|---|---|---|---|---|
| AWS Lambda + API GW | GB-sec @ $0.000016667 | $0.20/M Lambda + $1.00/M HTTP API calls | 1M req + 400k GB-s/mo (12 mo) + 1M API calls/mo | 15 min | 1,000 per region (can request more) |
| Modal | CPU-s @ $0.0000131 + GiB-s @ $0.00000222 | โ No per-request fee | $30/mo compute credits (Starter) | Func: 24h โฎ HTTP: 150s โ 303 redirect | Starter: 100 containers / 200 req/s โฎ Team: 1,000 containers |
Access Off-the-Shelf OS Models (via API):
| Platform/Tool | Released | GitHub |
|---|---|---|
| Together.ai | N/A | ๐ด |
| Fireworks.ai | N/A | |
| Replicate | 2019 | |
| Groq | N/A | |
| DeepInfra | N/A | |
| Bedrock | N/A | |
| Lepton | N/A | |
| Fal.ai | N/A | |
| VertexAI | N/A | |
Local Inference
| Framework | Browser Chat ๐ฅ๏ธ | Organization | Open Source | GitHub |
|---|---|---|---|---|
| Llama.cpp | โ | ggerganov | | |
| Ollama | โ | Ollama | | |
| gpt4all | โ | Nomic.ai | | |
| LMStudio | โ | LMStudio AI | ๐ด | |
| OpenLLM | โ | BentoML | | |
LLM Serving Frameworks
| Framework | Open Source | GitHub |
|---|---|---|
| vLLM | | |
| OpenLLM | | |
| TGI (Text Generation Inference) | | |
| TensorRT LLM | | |
| Ray Serve | | |
| LMDeploy | | |
| Ollama | | |
| MLC-LLM | | |
Building Open-Source LLM Web Chat UIs
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| Text Generation WebUI | oobabooga | A Gradio web UI for Large Language Models. | | |
| Jan AI | Jan HQ | An open source alternative to ChatGPT that runs 100% offline on your computer. Multiple engine support (llama.cpp, TensorRT-LLM). | | |
| AnythingLLM | Mintplex Labs | The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, and more. | | |
| Superagent | Superagent AI | Allows developers to add powerful AI assistants to their applications using LLMs and RAG. | | |
| Bionic-GPT | Bionic GPT | A ChatGPT replacement offering generative AI advantages while maintaining strict data confidentiality. | | |
| Open WebUI | Open WebUI | A user-friendly web interface for interacting with Large Language Models (LLMs). | | |
| Xyne | xynehq | A sleek, minimal web chat interface for interacting with Large Language Models. | | |
| Assistant UI | assistant-ui | An open-source ChatGPT-like interface with a clean and responsive design. | | |
| Scira | zaidmukaddam | An AI-powered search interface that leverages LLMs for intelligent search results. | | |
| Onyx | onyx-dot-app | A customizable and extendable web chat UI for interacting with large language models. | | |
| NextChat | ChatGPTNextWeb | A Next.js-based, open-source ChatGPT clone for seamless web interaction. | | |
Rent GPUs (Fine-Tuning, Deploying, Training)
| Platform | Templates | Beginner Friendly | GitHub |
|---|---|---|---|
| Brev.dev | Fine-tuning | โ | |
| Modal | Fine-tuning | โ | |
| Hyperbolic AI | None | โ | |
| RunPod | None | โ | |
| Paperspace | Fine-tuning | โ | |
| Colab | Small models only | โ | |
Fine-Tuning with No-Code UI
| Tool | Beginner Friendly | Open Source | GitHub |
|---|---|---|---|
| Together.ai | โ | โ | N/A |
| Hugging Face AutoTrain | โ | โ | |
| AutoML | โ | โ | |
| LLaMA-Factory | โ | โ | |
| H2O LLM Studio | โ | โ | |
Fine-Tuning Frameworks
OS Agentic/AI Workflow
| Framework | Open Source | Beginner Friendly | Released | GitHub |
|---|---|---|---|---|
| LangChain | | โ | 2022 | |
| LlamaIndex | | โ | 2023 | |
| Swarms | | โ | 2023 | |
| CrewAI | | โ | 2023 | |
| Autogen | | โ | 2023 | |
| AutoChain | | โ | 2023 | |
| SuperAGI | | โ | 2023 | |
| AILegion | | โ | 2023 | |
| MemGPT (Letta) | | โ | 2023 | |
| uAgents | | โ | 2023 | |
| AGiXT | | โ | 2023 | |
| Dify | | โ | 2024 | |
| TaskingAI | | โ | 2024 | |
| Bee Agent Framework | | โ | 2024 | |
| Swarms | | โ | 2024 | |
| IoA | | โ | 2024 | |
| Upsonic | | โ | 2024 | |
| Parlant | | โ | 2024 | |
| Rig | | โ | 2024 | |
| eliza | | โ | 2024 | |
| TensorZero | | โ | 2024 | |
| AgentDock | | โ | 2025 | |
Top Agentic Frameworks
| Framework | Open Source | Beginner Friendly | Released | GitHub |
|---|---|---|---|---|
| LangGraph | | โ | 2023 | |
| Flowise | | โ | 2023 | |
| Langroid | | โ | 2023 | |
| smolagents | | โ | 2024 | |
| Semantic Kernel | | โ | 2023 | |
| Atomic Agents | | โ | 2024 | |
| Agno | | โ | 2024 | |
| PydanticAI | | โ | 2024 | |
| Mastra | | โ | 2025 | |
Agentic Frameworks: Core Capabilities
| Framework | Memory & RAG | Multimodality | Multi-agent Support | Observability |
|---|---|---|---|---|
| AgentDock | Built-in RAG system; knowledge base integration | ๐ข Multi-modal (text, voice, tools, APIs) | Visual workflow orchestration & agent chains | Comprehensive LLM traceability & credit tracking |
| Agno | Integrated memory & vector DB/RAG | ๐ข Native (text, image, audio, video) | Supervisor-worker roles | Built-in cloud dashboard/logging |
| LangGraph | Persistent state; easy external integration | ๐ธ Primarily text; extendable via nodes | Hierarchical orchestration | LangSmith integration & graph editor |
| SmolAgents | Built-in short-term; custom long-term | ๐ธ Vision agents via VLMs | Modular multi-agent composition | Minimal; relies on external logging |
| Mastra | Persistent workflows; native RAG pipelines | ๐ข Multi-modal via integrations | Native multi-agent workflows | Built-in OpenTelemetry dashboards |
| Pydantic AI | DI-based memory & RAG integration | ๐ธ Text-first; multimodal via custom DI | Type-safe manual orchestration | Limited; Python logging/OpenTelemetry |
| Atomic Agents | Per-agent memory & RAG (vector DB) | ๐ข Native multi-modal | Explicit chaining of workflows | Minimal; external instrumentation recommended |
| Autogen | Short-term built-in; external long-term | ๐ธ Text-mainly; extensible | Emergent, free-form collaboration | Moderate; internal logging, no dashboard |
| CrewAI | Stateful memory & team-based RAG | ๐ข Diverse modalities (text, image, etc.) | Supervisor-led multi-team workflows | Integrated dashboards for logging & monitoring |
Please see this google sheet with more columns.
Visual AI Agent Builders
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| Rivet | Ironclad | A visual builder to design and deploy AI agent workflows. | | |
| PySpur | PySpur-Dev | A tool to build and visualize AI agents seamlessly. | | |
| Flowise | FlowiseAI | A noโcode, visual platform for designing AI agent workflows. | | |
๐ฌ Model Call Pricing for agent systems (Text-only (2000 tokens in, 100 token out), Flat Rate)
| Model | ๐ต $ / Call | ๐ฏ 100 Calls | ๐งฎ 1,000 Calls | ๐ 30,000 Calls |
|---|---|---|---|---|
| Gemini Flash 2.0 | $0.00000 | $0.02 | $0.24 | $7.20 |
| GPT-4o mini | $0.00144 | $0.14 | $1.44 | $43.20 |
| GPT-4.1 | $0.00480 | $0.48 | $4.80 | $144.00 |
| Gemini Pro 2.5 | $0.00350 | $0.35 | $3.50 | $105.00 |
| Claude Haiku 3.5 | $0.00200 | $0.20 | $2.00 | $60.00 |
| Claude Sonnet 4 | $0.00750 | $0.75 | $7.50 | $225.00 |
| GPT-4o | $0.01200 | $1.20 | $12.00 | $360.00 |
| OpenAI o3 | $0.02400 | $2.40 | $24.00 | $720.00 |
Agentic Tools (for โbuildingโ)
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| browser-use | browser-use | Integrates browser functionalities into agentic workflows. | | |
| code2prompt | mufeedvh | Converts code snippets into actionable prompts for development. | | |
| note-gen | codexu | Automatically generates notes and documentation from your code. | | |
| refly | refly-ai | Automates code refactoring and prompt generation tasks. | | |
| potpie | potpie-ai | A toolkit for prototyping and building AI agent pipelines. | | |
| AgentStack | AgentOps-AI | A comprehensive stack for constructing and deploying AI agents. | | |
| browser | lightpanda-io | A browserโbased tool designed for integrating agentic functionalities. | | |
| Memary | kingjulio8238 | A memory module for retaining context in agent workflows. | | |
| open-canvas | langchain-ai | A visual interface for designing agent workflows with LangChain. | | |
| agent-service-toolkit | JoshuaC215 | A toolkit for building and deploying agent-based services. | | |
Virtual Brains
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| Leon | leon-ai | An openโsource personal assistant and automation platform powered by AI. | | |
| Khoj | khoj-ai | A virtual brain for organizing and retrieving your knowledge using AI. | | |
AI Agents
| Framework | Organization | Open Source | Released | GitHub |
|---|---|---|---|---|
| GPT Engineer | GPT Engineer Org | | 2023 | |
| XAgent | OpenBMB | | 2023 | |
| Bolt.new | StackBlitz | | 2023 | |
| Goose | Block | | 2023 | |
| AI Hedge Fund | virattt | | 2023 | |
| FinRobot | AI4Finance Foundation | | 2024 | |
| STORM | Stanford OVAL | | 2024 | |
| Multion | MULTI-ON | ๐ด | N/A | |
| Minion | Minion AI | ๐ด | N/A | |
Long-Term Memory
| Provider | Community | Founded | GitHub | โญ Stars | Open Source |
|---|---|---|---|---|---|
| Letta | ๐ฌ Active dev community | Oct 2023 | | 17k | โ Apache-2.0 |
| Zep | ๐ค Moderate community | Aug 2024 | | 11.6k | โ ๏ธ Graphiti CE (Apache-2.0) |
| MemoRAG | ๐งช Small research group | Sep 2024 | | 1.8k | โ Apache-2.0 |
| Memary | ๐ง Niche community | April 2024 | | 2.3k | โ MIT |
| Cognee | ๐ Moderate | Aug 2023 | | 5.8k | โ Apache-2.0 |
| Mem0 | ๐ Fast-growing | June 2023 | | 35.2k | โ Apache-2.0 |
Memory Features Comparison
| Provider | Based | Optional KG | Self-Editing / Agentic | Rolling Summaries | Categories |
|---|---|---|---|---|---|
| Letta | ๐งฎ Vector | โ ๏ธ Partial | โ Yes | โ ๏ธ Partial (memory blocks) | โ Yes |
| Zep | ๐ง KG | - | โ Yes | โ Auto chat summarization | โ Yes |
| MemoRAG | ๐งฎ Vector | โ No | โ Yes | โ Uses long-range model | โ No |
| Memary | ๐ง KG | - | โ Yes | โ ๏ธ Plans โrewindโ feature | โ Yes |
| Cognee | ๐ง KG | - | โ Yes | โ No auto summaries | โ Yes |
| Mem0 | ๐งฎ Vector | โ No | โ Yes | โ Not explicit | โ Yes |
Enterprise Security (Cloud-Based Use)
| Provider | Enterprise Security |
|---|---|
| Mem0 | ๐ Hosted with encryption, org/project roles, GDPR-friendly delete. Uses Graphlit (SOC 2 not stated). |
| Letta | โ๏ธ Self-hosted or managed server. User auth & ID-partitioned memory. Graphlit-based. No public SSO details. |
| Zep | โ SOC 2 Type 2. Encrypted at rest/in transit, access controls, JWT, and deletion API ("Right to be Forgotten"). |
| MemoRAG | ๐ Self-host |
| Memary | ๐ Self-host |
| Cognee | ๐ Self-host |
Pricing by Monthly Message Volume
| Provider | 1K msgs/mo | 10K msgs/mo | 100K msgs/mo | 1M msgs/mo |
|---|---|---|---|---|
| Mem0 | ๐ Free | ๐ Freeโ$29 | ๐ต $249 | ๐ข Enterprise (custom) |
| Letta | ๐ Free | ๐ต $20 | ๐ต $750 | ๐ข Enterprise (custom) |
| Zep | ๐ Free | ๐ Free | ๐ต ~ $112.50 | ๐ต ~ $1,237 |
| MemoRAG | ๐ป GPU Server (~$150โ300/mo) | ๐ป GPU Server (~$150โ300/mo) | ๐ป Multi-GPU ($500+) | ๐ฅ๏ธ Cluster ($1K+/mo) |
| Self-host | ๐ฅ๏ธ Small VM (~$15/mo) | ๐ฅ๏ธ Small VM (~$15โ20/mo) | ๐ฅ๏ธ Medium VM ($50โ$100/mo) | ๐ฅ๏ธ Large VM ($200+/mo) |
Evaluation Frameworks and add-ons
| Framework | Open Source | Beginner Friendly | Released | GitHub |
|---|---|---|---|---|
| TruLens | | โ | 2023 | |
| Promptfoo | | โ | 2023 | |
| DeepEval | | โ | 2024 | |
| RAGAS | | โ | 2023 | |
| OpenAI Evals | | โ | 2023 | |
| LangChain OpenEvals | | โ | 2025 | |
| LangChain AgentEvals | | โ | 2025 | |
| LlamaIndex Eval | | โ | 2023 | |
Evaluation Frameworks: Core Differences
| Framework | Pytest / CLI Runner | Metrics Ready-made | Synthetic Data Gen | Offline Judge | Model-Agnostic | Safety | Red-Team | Custom Metrics (setup speed) |
|---|---|---|---|---|---|---|---|---|
| DeepEval | ๐ข deepeval test | 40 + | ๐ข deepeval create-dataset | ๐ข | ๐ข | ๐ข | ๐ข | ๐ข G-Eval builder โ minutes (one function) |
| RAGAS | โ (script asserts) | 6 core RAG + ๐ธ | ๐ข KG-based Q-gen | ๐ข | ๐ข | ๐ธ DIY | โ | ๐ข AspectCritic one-liner โ minutes |
| MLflow Evaluate | โ (mlflow.evaluate) | 3-4 | โ BYO | ๐ธ possible | ๐ธ | ๐ข | โ | ๐ข Subclass scorer โ few lines, ~hour |
| OpenAI Evals | ๐ข CLI orchestrator | ~10 templates | ๐ธ helper script | โ | ๐ข | โ | โ | ๐ข Full Python/YAML eval โ flexible but slower |
Co-Pilots
Voice API
| Framework | Open Source | GitHub |
|---|---|---|
| VAPI.ai | ๐ด | |
| Bland.ai | ๐ด | N/A |
| CallAnnie | ๐ด | N/A |
| RealtimeTTS | | |
| RealtimeSTT | | |
| Coqui TTS | | |
Open Source TTS Models
| Model | License | Stars/Likes | Downloads (Last Month) | Repository |
|---|---|---|---|---|
| Kokoro-82M | Apache 2.0 | โญ 3.16k (HF) | ๐ฅ 557,392 | Hugging Face |
| Zonos-v0.1-transformer | Apache 2.0 | โญ 249 (HF) | ๐ฅ 24,240 | Hugging Face |
| XTTS-v2 | Non-Commercial | โค๏ธ 368 (HF) | ๐ฅ 2,545,850 | Hugging Face |
| ChatTTS | AGPL-3.0 | N/A | N/A | GitHub |
| MeloTTS | MIT | N/A | N/A | GitHub |
For more TTS models and rankings, check out the TTS Leaderboard.
LLM Application Frameworks
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| Eino | CloudWeGo | A lightweight LLM application framework for scalable AI solutions. | | |
| Conversation Knowledge Mining Solution Accelerator | Microsoft | A solution accelerator for integrating conversation intelligence and knowledge mining using LLMs. | | |
| Olmocr | AllenAI | An OCR framework optimized for integration with language models. | | |
| PDFMathTranslate | Byaidu | A tool for converting and translating mathematical content in PDFs using LLMs. | | |
| Podcastfy | souzatharsis | A tool to generate podcasts from written content using LLMs. | | |
| Pandas AI | sinaptik-ai | Brings LLM-powered analytics to pandas dataframes. | | |
| Ramalama | containers | An LLM application framework for containerized deployment of AI solutions. | | |
| Robyn | facebookexperimental | A scalable framework for building LLM applications from Facebook Experimental. | | |
| ExtractThinker | enoch3712 | A tool for extracting and synthesizing insights from textual data using LLMs. | | |
OS RAG Frameworks
| Framework | Organization | Open Source | Released | GitHub |
|---|---|---|---|---|
| Haystack | deepset.ai | | 2023 | |
| RAGflow | Infiniflow | | 2024 | |
| txtai | Neuml | | 2022 | |
| LLM App | Pathway | | 2023 | |
| Cognita | Truefoundry | | 2024 | |
| R2R | SciPhi AI | | 2024 | |
| Raptor | Parth Sarthi | | 2024 | |
| LightRAG | HKUDS | | 2023 | |
| PIKE-RAG | Microsoft | | 2024 | |
| KAG | OpenSPG | | 2024 | |
| MemoRAG | qhjqhj00 | | 2023 | |
See RAG_Techniques if you get stuck (not always needed)
๐ Vector DBs โ FOSS, Performance, Pricing, DevX
| Vector DB | License | โก Perf / Throughput | โฑ๏ธ Latency (Real-World) | โ๏ธ Cloud Pricing / Free Tier | ๐ป Dev Experience |
|---|---|---|---|---|---|
| Qdrant | Apache 2.0 | ๐ฅ Highest RPS, lowest latency in single-node bench (โฅ4ร vs prev. run) | p95 < 10ms for 1M vecs (1 thread) | Always-on 1GB free; pay-go โ $0.014/hr | REST + gRPC; 7 lang clients; filter-aware HNSW; hybrid support; Python โembeddedโ mode |
| Milvus / Zilliz Cloud | Apache 2.0 | ๐ Fastest index build; RPS trails Qdrant for high-dim vecs | p95 โ 10โ20ms @ 1M 768-dim (DiskANN, vendor data) | 5GB free; serverless $0.30/GB-mo; dedicated from $99/mo | New SDK v2 (async, schema cache); Python/Go/Java/Node support |
| Weaviate | BSD-3-Clause | โ๏ธ Least bench gains, but decent recall (95%+) and throughput | โLow-msโ claimed; users report 100โ300ms if misconfigured | Starts $25/mo; 14-day sandbox free | GraphQL + REST; strong SDKs (Py/TS/Go/Java); easy RAG + hybrid templates |
| pgvector | MIT | ๐ฅ 28ร lower p95 & 16ร higher QPS vs Pinecone s1 @ 99% recall (50M) | p95 < 50ms @ 50M 768-dim (Timescale test) | Neon/Supabase offer free Postgres with pgvector (0.5โ1GB, ~200h CPU) | Pure SQL; supports joins + ACID; great for hybrid text + dense queries |
| Redis 8 Vector | AGPLv3 / RSAL / SSPL | ๐งต 3.4ร higher QPS vs Qdrant, 4ร vs Milvus @ โฅ0.98 recall | Sub-ms avg, <10ms under load (vendor); 9.7ร lower than Aurora+pgvector | Redis Cloud: 30MB free, pay-go from $5/mo; Flex $0.007/hr | Redis Vector Library + RAG helpers; OM clients for .NET/Py/JS; fast setup |
๐พ Vector DB Cloud Pricing (2000-char Chunks, ~768-dim)
| # Chunks | Data Size | ๐ฃ Milvus / Zilliz Cloud (Serverless) | ๐ข Qdrant Cloud | ๐ก Weaviate Cloud (โStandardโ) |
|---|---|---|---|---|
| 10k | ~0.07 GB | ๐ Free โ within 5 GB tier | ๐ Free โ fits in 1 GB RAM / 4 GB disk | $25 base + $1.2 dim fee โ $26 |
| 100k | ~0.67 GB | ๐ Still under 5 GB | ๐ Fits with compression in 4 GB disk | $25 + $12.0 dim fee โ $37 |
| 1M | ~6.7 GB | ๐ต โ $2 storage; add vCU fees or $99 dedicated | ๐ต Needs 10 GB cluster โ โ $20/mo | $25 + $120.5 dim fee โ $145 |
| 10M | ~67 GB | ๐ต โ $20 storage; + compute: $100โ150 total | ๐ต Needs 64+ GB โ $120โ150/mo estimate | $25 + $1,204 dim fee โ $1,230 |
๐ง Embedding Cost โ OpenAI (Small Model, per Chunk Size)
| # Chunks | ๐ 1,000 Chars | ๐ 2,000 Chars | ๐ 3,000 Chars |
|---|---|---|---|
| 1,000 | $0.01 | $0.01 | $0.01 |
| 10,000 | $0.05 | $0.10 | $0.15 |
| 100,000 | $0.50 | $1.00 | $1.50 |
| 1,000,000 | $5.00 | $10.00 | $15.00 |
| 10,000,000 | $50.00 | $100.00 | $150.00 |
AI Tools (for โusingโ)
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| magic-resume | JOYCEQL | An AI-powered tool for generating resumes. | | |
| VideoCaptioner | WEIFENG2333 | An AI tool for automatically generating video captions. | | |
| DeepSeekAI | DeepLifeStudio | Browser extension for invoking the DeepSeek AI large model. | | |
| logocreator | Nutlope | A tool for creating logos using AI. | | |
| blinkshot | Nutlope | An AI-powered tool for capturing and enhancing screenshots. | | |
| pollinations | pollinations | A tool for generating creative images and artwork using AI. | | |
| PromptWizard | microsoft | A tool to generate, manage, and optimize prompts for AI models. | | |
| Open-Interface | AmberSahdev | Control Any Computer Using LLMs. | | |
| wut | shobrook | LLM for the terminal | | |
Training/Optimization
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| transformerlab-app | transformerlab | An application for training and optimizing transformer models. | | |
| fluxgym | cocktailpeanut | A gym environment for reinforcement learning training and optimization. | | |
| AutoGPTQ | AutoGPTQ | A tool for automating GPT quantization and optimization. | | |
AI Models
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| WALDO | stephansturges | An AI model for visual reasoning and object detection. | | |
| Janus | deepseek-ai | A multi-modal AI model for advanced data processing. | | |
| ModernBERT | AnswerDotAI | A modernized version of BERT for natural language processing tasks. | | |
| Magma | microsoft | A scalable AI model for large-scale data analysis. | | |
| Cosmos-Nemotron | NVlabs | An AI model for advanced image and video processing. | | |
| Paints-UNDO | lllyasviel | An interactive AI model for image generation and editing. | | |
Monitoring
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| helicone | Helicone | A platform for monitoring and analyzing AI model performance. | | |
| langwatch | langwatch | A tool for monitoring outputs and performance of language models. | | |
Infrastructure
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| gpustack | gpustack | A toolkit for managing GPU infrastructure for AI workloads. | | |
| harbor | av | A repository for containerized AI infrastructure management. | | |
Research Papers on Chain-of-Thought Prompting
| Publication Date | Title | ๐ | Authors | Organization | Technique |
|---|---|---|---|---|---|
| January 28, 2022 | Chain-of-Thought Prompting Elicits Reasoning in Large Language Models | ๐ | Jason Wei, et al. | DeepMind | CoT Prompting |
| March 21, 2022 | Self-Consistency Improves Chain of Thought Reasoning in Language Models | ๐ | Xuezhi Wang et al. | DeepMind | CoT with Self-Consistency |
| May 21, 2022 | Least-to-Most Prompting Enables Complex Reasoning in Large Language Models | ๐ | Denny Zhou et al. | DeepMind | Least-to-Most Prompting |
| May 21, 2022 | Large Language Models are Zero-Shot Reasoners | ๐ | Takeshi Kojima, et al. | DeepMind | Zero-shot-CoT |
| October 6, 2022 | ReAct: Synergizing Reasoning and Acting in Language Models | ๐ | Shunyu Yao et al. | Princeton University | ReAct |
| April 1, 2023 | Teaching Large Language Models to Self-Debug | ๐ | Xiang Lisa Li, et al. | DeepMind, Stanford University | Self-Debugging |
| May 6, 2023 | Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models | ๐ | Lei Wang et al. | The Chinese University of Hong Kong, SenseTime Research | Plan-and-Solve Prompting |
| May 23, 2023 | Letโs Verify Step by Step | ๐ | Anya Goyal, et al. | DeepMind | Verification for CoT |
| October 3, 2023 | Large Language Models Cannot Self-Correct Reasoning Yet | ๐ | Qingxiu Dong, et al. | The Chinese University of Hong Kong, Huawei Noah's Ark Lab | Self-Correction in LLMs |
| November 2023 | Universal Self-Consistency for Large Language Model Generation | ๐ | Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, Denny Zhou | DeepMind | Universal Self-Consistency |
| May 17, 2023 | Tree of Thoughts: Deliberate Problem Solving with Large Language Models | ๐ | Shunyu Yao, et al. | Princeton University, DeepMind | Tree-of-Thought |
| February 15, 2024 | Chain-of-Thought Reasoning Without Prompting | ๐ | Xuezhi Wang, Denny Zhou | DeepMind | Chain-of-Thought Decoding |
| March 21, 2024 | ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting | ๐ | Xiaoxue Cheng et al. | Renmin University of China | CoTGenius |
| June 2024 | Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models | ๐ | Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, Yu-Xiong Wang | Language Agent Tree Search (LATS) | |
| May 2024 | Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning | ๐ | Yuxi Xie, et al. | National University of Singapore, DeepMind | MCTS |
| September 18, 2024 | To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning | ๐ | Zayne Sprague, et al. | The University of Texas at Austin, Johns Hopkins University, Princeton University | Meta-analysis of CoT |
| September 25, 2024 | Chain-of-Thoughtlessness? An Analysis of CoT in Planning | ๐ | Kaya Stechly, et al. | Arizona State University | Analysis of CoT in Planning |
| October 18, 2024 | Supervised Chain of Thought | ๐ | Xiang Zhang, Dujian Ding | University of British Columbia | Supervised Chain of Thought |
| October 24, 2024 | On examples: A Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstration | ๐ | Zhiqiang Hu, et al. | Amazon, Michigan State University | Theoretical Analysis of CoT |
CoT Implementations
| Implementation | Link | Author | GitHub Stars | GitHub Followers |
|---|---|---|---|---|
| CoT | chain-of-thought-hub | Franx Yao | | |
| CoT | optillm | Codelion | | |
| CoT | auto-cot | Amazon Science | | |
| CoT | g1 | BKlieger Groq | | |
| Decoding CoT | optillm/cot_decoding.py | Codelion | | |
| Tree of Thoughts | tree-of-thought-llm | Princeton NLP | | |
| Tree of Thoughts | tree-of-thoughts | Kye Gomez | | |
| Tree of Thoughts | saplings | Shobrook | | |
| MCTS | optillm/mcts.py | Codelion | | |
| Graph of Thoughts | graph-of-thoughts | SPCL | | |
| Other | CPO | SAIL SG | | |
| Other | Everything-of-Thoughts-XoT | Microsoft | | |
CoT Fine-Tuned Models & Datasets
Models
| Model Name | Author | Size | Link |
|---|---|---|---|
| CoT-T5-3B | KAIST AI | 3B | ๐ |
| CoT-T5-11B | KAIST AI | 11B | ๐ |
| Llama-3.2V-11B-cot | Xkev | 11B | ๐ |
| Llama-3.1-8B-Instruct-Reasoner-1o1_v0.3 | Lyte | 8B | ๐ |
Datasets
| Dataset Name | Author | Data Size | Likes | Link |
|---|---|---|---|---|
| chain-of-thought-sharegpt | Isaiah Bjork | 7.14k rows | ๐ 8 | ๐ |
| CoT-Collection | KAIST AI | 1.84 million rows | ๐ 122 | ๐ |
| Reasoner-1o1-v0.3-HQ | Lyte | 370 rows | ๐ 7 | ๐ |
| OpenLongCoT-Pretrain | qq8933 | 103k rows | ๐ 86 | ๐ |
Learning Resources
| Tool | Organization | Description | Open Source | GitHub |
|---|---|---|---|---|
| awesome-cursorrules | PatrickJS | A curated list of resources and guides on cursorrules. | | |
| ai-engineering-hub | patchy631 | A hub of AI engineering learning resources, tutorials, and best practices. | | |
| GenAI_Agents | NirDiamant | Resources and examples for building Generative AI Agents. | | |
| learn-agentic-ai | panaversity | Learning materials for understanding and building agentic AI. | | |
| awesome-generative-ai | steven2358 | A curated list of generative AI resources and projects. | | |
| awesome-mcp-servers | punkpeye | A curated collection of awesome MCP servers resources. | | |
| GenAI-Showcase | mongodb-developer | A showcase of innovative Generative AI projects. | | |
| well-architected-iac-analyzer | aws-samples | A tool to analyze and ensure well-architected Infrastructure as Code practices. | | |
| llama-cookbook | meta-llama | A collection of recipes and guides for working with LLaMA models. | | |
| optillm | codelion | Resources for optimizing LLM usage and performance. | | |
| cursor.directory | pontusab | A directory of tools and resources related to cursor-based workflows. | | |
| GenAI_Agents | NirDiamant | A curated collection of generative AI agents and related tools. | | |
large-language-models
llm
resources
Similar repos
Contributors
Languages
Python
100.0%