Sumanth077/ai-engineering-toolkit

A curated list of 100+ libraries and frameworks for AI engineers building with LLMs

3,396

32 commits

updated May 11, 2026

aiengineering.beehiiv.com

See the code

README

AI Engineering Toolkit🔥

Build better LLM apps — faster, smarter, production-ready.

A curated, list of 100+ libraries and frameworks for AI engineers building with Large Language Models. This toolkit includes battle-tested tools, frameworks, templates, and reference implementations for developing, deploying, and optimizing LLM-powered systems.

🛠️ Tooling for AI Engineers

Vector Databases

Tool	Description	Language	License
Pinecone	Managed vector database for production AI applications	API/SDK	Commercial
Weaviate	Open-source vector database with GraphQL API	Go	BSD-3
Qdrant	Vector similarity search engine with extended filtering	Rust	Apache-2.0
Chroma	Open-source embedding database for LLM apps	Python	Apache-2.0
Milvus	Cloud-native vector database for scalable similarity search	Go/C++	Apache-2.0
FAISS	Library for efficient similarity search and clustering	C++/Python	MIT
Deep Lake	AI-native data lake with versioned datasets, optimized for embeddings and multimodal storage	Python	Apache-2.0
Vectara	Managed RAG platform with APIs for retrieval and generation	Python/Go	Commercial

Orchestration & Workflows

Tool	Description	Language	License
LangChain	Framework for developing LLM applications	Python/JS	MIT
LlamaIndex	Data framework for LLM applications	Python	MIT
Haystack	End-to-end NLP framework for production	Python	Apache-2.0
DSPy	Framework for algorithmically optimizing LM prompts	Python	MIT
Semantic Kernel	SDK for integrating AI into conventional programming languages	C#/Python/Java	MIT
Langflow	Visual no-code platform for building and deploying LLM workflows	Python/TypeScript	MIT
Flowise	Drag-and-drop UI for creating LLM chains and agents	TypeScript	MIT
Promptflow	Workflow orchestration for LLM pipelines, evaluation, and deployment	Python	MIT
Dify	Production-ready open source platform combining RAG pipelines, agent capabilities, model management, and observability in one UI	TypeScript/Python	Apache-2.0

PDF Extraction Tools

Tool	Description	Language	License
Docling	AI-powered toolkit converting PDF, DOCX, PPTX, HTML, images into structured JSON/Markdown with layout, OCR, table, and code recognition	Python	MIT
pdfplumber	Drill through PDFs at a character level, extract text & tables, and visually debug extraction	Python	MIT
PyMuPDF (fitz)	Lightweight, high-performance PDF parser for text/image extraction and manipulation	Python / C	AGPL-3.0
PDF.js	Browser-based PDF renderer with text extraction capabilities	JavaScript	Apache-2.0
Camelot	Extracts structured tabular data from PDFs into DataFrames and CSVs	Python	MIT
Unstructured	Parse PDFs, DOCX, HTML into structured JSON for LLM workflows	Python	Apache-2.0
pdfminer.six	Detailed PDF text extraction and layout analysis	Python	MIT
Llama Parse	Structured parsing of PDFs and documents optimized for LLMs	Python	Apache-2.0
MegaParse	Universal parser for PDFs, HTML, and semi-structured documents	Python	Apache-2.0
ExtractThinker	Intelligent document extraction framework with schema mapping	Python	MIT
PyMuPDF4LLM	Wrapper around PyMuPDF for LLM-ready text, tables, and image extraction	Python	Apache-2.0

RAG (Retrieval-Augmented Generation)

Tool	Description	Language	License
RAGFlow	Open-source RAG engine based on deep document understanding	Python	Apache-2.0
Verba	Retrieval Augmented Generation (RAG) chatbot	Python	BSD-3
PrivateGPT	Interact with documents using local LLMs	Python	Apache-2.0
AnythingLLM	All-in-one AI application for any LLM	JavaScript	MIT
Quivr	Your GenAI second brain	Python/TypeScript	Apache-2.0
Jina	Cloud-native neural search framework for multimodal RAG	Python	Apache-2.0
txtai	All-in-one embeddings database for semantic search and workflows	Python	Apache-2.0
FastGraph RAG	Graph-based RAG framework for structured retrieval	Python	MIT
Chonkie	Chunking utility for efficient document processing in RAG	Python	-
FlashRAG	Low-latency RAG research toolkit with modular design and benchmarks	Python	-
Llmware	Lightweight framework for building RAG-based apps	Python	Apache-2.0

Evaluation & Testing

Tool	Description	Language	License
Evals	OpenAI's framework for creating and running LLM evaluations	Python	MIT
Ragas	Evaluation framework for RAG pipelines	Python	Apache-2.0
Opik	DevOps platform for evaluation, monitoring, and observability	Python	Apache-2.0
Phoenix	ML observability for LLM, vision, language, and tabular models	Python	Apache-2.0
DeepEval	LLM evaluation framework for unit testing LLM outputs	Python	Apache-2.0
TruLens	Evaluation and tracking for LLM experiments	Python	MIT
UpTrain	Open-source tool to evaluate and improve LLM applications	Python	Apache-2.0
Giskard	Testing framework for ML/LLMs with bias and robustness checks	Python	Apache-2.0
Weave	Experiment tracking, debugging, and logging for LLM workflows	Python	Apache-2.0
Lighteval	Lightweight and fast evaluation framework from Hugging Face	Python	Apache-2.0
Langfuse	Open source LLM engineering platform with tracing, evals, prompt management, and metrics	TypeScript/Python	MIT
Helicone	Open source observability and monitoring platform for debugging and improving LLM apps	TypeScript	Apache-2.0
OpenLLMetry	OpenTelemetry-based observability extensions for LLM apps — plugs into Datadog, Honeycomb, and more	Python	Apache-2.0

Model Management

Tool	Description	Language	License
Hugging Face Hub	Client library for Hugging Face Hub	Python	Apache-2.0
MLflow	Platform for ML lifecycle management	Python	Apache-2.0
Weights & Biases	Developer tools for ML	Python	MIT
DVC	Data version control for ML projects	Python	Apache-2.0
ClearML	End-to-end MLOps platform with LLM support	Python	Apache-2.0

Data Collection & Web Scraping

Tool	Description	Language	License
Firecrawl	AI-powered web crawler that extracts and structures content for LLM pipelines	TypeScript	MIT
Scrapy	Fast, high-level web crawling & scraping framework	Python	BSD-3
Playwright	Web automation & scraping with headless browsers	TypeScript/Python/Java/.NET	Apache-2.0
BeautifulSoup	Easy HTML/XML parsing for quick scraping tasks	Python	MIT
Selenium	Browser automation framework (supports scraping)	Multiple	Apache-2.0
Newspaper3k	News & article extraction library	Python	MIT
Crawl4AI	Fast, lightweight, and modern web crawling & scraping library for AI data pipelines	Python	Apache-2.0
Colly	High-performance scraping framework for Go	Go	BSD-2
Trafilatura	Extract clean text from web pages for LLM training corpora	Python	MIT
ScrapeGraphAI	Use LLMs to extract structured data from websites and documents	Python	MIT
Crawlee	Web scraping and crawling framework for large-scale data collection	TypeScript	Apache-2.0

🤖 Agent Frameworks

Framework	Description	Language	License
Google's ADK	Flexible and modular framework for developing and deploying AI agents	Python / Java	Apache-2.0
AutoGen	Multi-agent conversation framework	Python	CC-BY-4.0
CrewAI	Framework for orchestrating role-playing autonomous AI agents	Python	MIT
LangGraph	Build resilient language agents as graphs	Python	MIT
AgentOps	Python SDK for AI agent monitoring, LLM cost tracking, benchmarking	Python	MIT
Swarm	Educational framework for exploring ergonomic, lightweight multi-agent orchestration	Python	MIT
Agency Swarm	An open-source agent framework designed to automate your workflows	Python	MIT
Multi-Agent Systems	Research into multi-agent systems and applications	Python	MIT
Auto-GPT	Autonomous AI agent for task execution using GPT models	Python	MIT
BabyAGI	Task-driven autonomous agent inspired by AGI	Python	MIT
SuperAGI	Infrastructure for building and managing autonomous agents	Python	MIT
Griptape	Framework for building AI agents with structured pipelines and memory	Python	Apache-2.0
Letta (MemGPT)	Long-term memory management for LLM agents	Python	MIT
Agno	Framework for building AI agents with RAG, workflows, and memory	Python	Apache-2.0
Agents SDK	SDK from Vercel for building agentic workflows and applications	TypeScript	Apache-2.0
Smolagents	Lightweight agent framework from Hugging Face	Python	Apache-2.0
Pydantic AI	Agent framework built on Pydantic for structured reasoning	Python	MIT
CAMEL	Multi-agent framework enabling role-play and collaboration	Python	Apache-2.0
Swarms	Enterprise agent orchestration framework (“Agency Swarm”)	Python	MIT
Langroid	Framework for building multi-agent conversational systems	Python	Apache-2.0
Upsonic	Agent framework focused on context management and tool use	Python	Apache-2.0
Mem0	Universal memory layer for AI agents — persistent, personalized memory across sessions	Python	Apache-2.0

📦 LLM Development & Optimization

LLM Training and Fine-Tuning

Tool	Description	Language	License
PyTorch Lightning	High-level PyTorch interface for LLMs	Python	Apache-2.0
unsloth	Fine-tune LLMs faster with less memory	Python	Apache-2.0
Axolotl	Post-training pipeline for AI models	Python	Apache-2.0
LLaMA-Factory	Easy & efficient LLM fine-tuning	Python	Apache-2.0
PEFT	Parameter-Efficient Fine-Tuning library	Python	Apache-2.0
DeepSpeed	Distributed training & inference optimization	Python	MIT
TRL	Train transformer LMs with reinforcement learning	Python	Apache-2.0
Transformers	Pretrained models for text, vision, and audio tasks	Python	Apache-2.0
LitGPT	Train and fine-tune LLMs lightning fast	Python	Apache-2.0
Ludwig	Low-code framework for custom LLMs	Python	Apache-2.0
xTuring	Fast fine-tuning of open-source LLMs	Python	Apache-2.0
RL4LMs	RL library to fine-tune LMs to human preferences	Python	Apache-2.0
torchtune	PyTorch-native library for fine-tuning LLMs	Python	BSD-3
Accelerate	Library to easily train on multiple GPUs/TPUs with mixed precision	Python	Apache-2.0

Open Source LLM Inference

Tool	Description	Language	License
LLM Compressor	Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment	Python	Apache-2.0
LightLLM	Lightweight Python-based LLM inference and serving framework with easy scalability and high performance	Python	Apache-2.0
vLLM	High-throughput and memory-efficient inference and serving engine for LLMs	Python	Apache-2.0
TensorRT-LLM	NVIDIA library for optimizing LLM inference with TensorRT	C++/Python	Apache-2.0
WebLLM	High-performance in-browser LLM inference engine	TypeScript/Python	Apache-2.0
SkyPilot	Unified framework to run ML workloads and LLMs on any cloud (AWS, GCP, Azure, Lambda, etc.) with auto-spot, data syncing, and cost optimization.	Python	Apache-2.0

LLM Safety and Security

Tool	Description	Language	License
Guardrails	Add guardrails to large language models	Python	MIT
NeMo Guardrails	Toolkit for adding programmable guardrails to LLM conversational systems	Python	Apache-2.0
Garak	LLM vulnerability scanner	Python	MIT
DeepTeam	LLM red teaming framework	Python	Apache-2.0

AI App Development Frameworks

Tool	Description	Language	License
Reflex	Build full-stack web apps powered by LLMs with Python-only workflows and reactive UIs.	Python	Apache-2.0
Gradio	Create quick, interactive UIs for LLM demos and prototypes.	Python	Apache-2.0
Streamlit	Build and share AI/ML apps fast with Python scripts and interactive widgets.	Python	Apache-2.0
Taipy	End-to-end Python framework for building production-ready AI apps with dashboards and pipelines.	Python	Apache-2.0
AI SDK UI	Vercel’s AI SDK for building chat & generative UIs	TypeScript	Apache-2.0
Simpleaichat	Minimal Python interface for prototyping conversational LLMs	Python	MIT
Chainlit	Framework for building and debugging LLM apps with a rich UI	Python	Apache-2.0
Mirascope	Lightweight Python toolkit for building LLM apps with Pydantic-native structured outputs and no extra abstractions	Python	MIT

Local Development & Serving

Tool	Description	Language	License
Ollama	Get up and running with large language models locally	Go	MIT
LM Studio	Desktop app for running local LLMs	-	Commercial
GPT4All	Open-source chatbot ecosystem	C++	MIT
LocalAI	Self-hosted OpenAI-compatible API	Go	MIT
llama.cpp	Lightweight, high-performance inference engine for running LLMs locally across CPU, GPU, and mobile backends	C++	MIT
LiteLLM	Lightweight OpenAI-compatible gateway for multiple LLM providers	Python	MIT
AI Gateway	Gateway for managing LLM requests, caching, and routing	Python	Apache-2.0
Langcorn	Serve LangChain applications via FastAPI with production-ready endpoints	Python	MIT
LitServe	High-speed GPU inference server with autoscaling and batch support	Python	Apache-2.0

Structured Generation

Tool	Description	Language	License
Instructor	Multi-language library for extracting structured, validated outputs from LLMs using Pydantic	Python/TS/Go	MIT
Outlines	Guarantees 100% valid structured outputs (JSON, regex, grammars) by constraining LLM token generation	Python	Apache-2.0
Guidance	Microsoft's language for controlling LLM outputs — interleave generation with conditionals, loops, and constraints	Python	MIT

LLM Inference Platforms

Platform	Description	Pricing	Features
Clarifai	Lightning-fast compute for AI models & agents	Free tier + Pay-as-you-go	Pre-trained models, Deploy your own models on Dedicated compute, Model training, Workflow automation
Modal	Serverless platform for AI/ML workloads	Pay-per-use	Serverless GPU, Auto-scaling
Replicate	Run open-source models with a cloud API	Pay-per-use	Pre-built models, Custom training
Together AI	Cloud platform for open-source models	Various	Open models, Fine-tuning
Anyscale	Ray-based platform for AI applications	Enterprise	Distributed training, Serving
Groq	Ultra-fast LPU inference engine for running open AI models	Free tier + Pay-per-use	Blazing fast speed, High throughput, Low latency, Open model support
OpenRouter	Universal API to find and route to the best LLMs from various providers	Free tier + Pay-per-use	Multi-provider access, Unified API, Model comparison, Caching
RouteLLM	Dynamic router for selecting best LLMs based on cost & performance	Open-source	Cost optimization, Multi-LLM routing

🤝 Contributing

We welcome contributions! This toolkit grows stronger with community input.

How to Contribute

Fork the repository
Create a feature branch (git checkout -b feature/amazing-tool)
Add your contribution (new tool, template, or tutorial)
Submit a pull request

Contribution Guidelines

Quality over quantity - Focus on tools and resources that provide real value
Production-ready - Include tools that work in real-world scenarios
Well-documented - Provide clear descriptions and usage examples
Up-to-date - Ensure tools are actively maintained

📧 Stay Connected

Get weekly AI engineering insights, tool reviews, and exclusive demos and AI Projects delivered to your inbox:

📧 Subscribe to AI Engineering Newsletter →

Join 100,000+ engineers building better LLM applications

Built with ❤️ for the AI Engineering community

Star ⭐ this repo if you find it helpful!

Significant stargazers

Arindam Majumder

894 followers · starred Aug 2025

Elijas Dapšauskas

219 followers · starred Aug 2025

Keyu Chen

12 followers · starred Aug 2025

Niranjan Uma Shankar

12 followers · starred Jul 2026

Similar repos

KalyanKS-NLP/llm-engineer-toolkit

A curated list of 120+ LLM libraries category wise.

10,886

break-into-data/ai-engineer-toolkit

Projects & Resources to help you become a better AI Developer.

2,192

uhub/awesome-llm

A curated list of awesome LLM frameworks, libraries and software.

860

SylphAI-Inc/LLM-engineer-handbook

A curated list of Large Language Model resources, covering model training, serving, fine-tuning,…

5,041

hades217/awesome-ai

A curated list of artificial intelligence resources (Courses, Tools, App, Open Source Project)

554

dmarx/awesome-llm-utilities

Tooling and frameworks to support building tools that utilize LLM agents

a16z-infra/llm-app-stack

1,321

fahimar/awesome-ai-engineering

Curated AI Agents, Automation & Engineering resources — each with a concrete first step on how to…