LLM-engineer-handbook

🔥 Large Language Models(LLM) have taken the NLP community AI community the Whole World by storm.

Why do we create this repo?

• Everyone can now build an LLM demo in minutes, but it takes a real LLM/AI expert to close the last mile of performance, security, and scalability gaps.
• The LLM space is complicated! This repo provides a curated list to help you navigate so that you are more likely to build production-grade LLM applications. It includes a collection of Large Language Model frameworks and tutorials, covering model training, serving, fine-tuning, LLM applications & prompt optimization, and LLMOps.

However, classical ML is not going away. Even LLMs need them. We have seen classical models used for protecting data privacy, detecing hallucinations, and more. So, do not forget to study the fundamentals of classical ML.

Overview

The current workflow might look like this: You build a demo using an existing application library or directly from LLM model provider SDKs. It works somehow, but you need to further create evaluation and training datasets to optimize the performance (e.g., accuracy, latency, cost).

You can do prompt engineering or auto-prompt optimization; you can create a larger dataset to fine-tune the LLM or use Direct Preference Optimization (DPO) to align the model with human preferences. Then you need to consider the serving and LLMOps to deploy the model at scale and pipelines to refresh the data.

We organize the resources by (1) tracking all libraries, frameworks, and tools, (2) learning resources on the whole LLM lifecycle, (3) understanding LLMs, (4) social accounts and community, and (5) how to contribute to this repo.

• LLM-engineer-handbook
  • Overview
• Libraries & Frameworks & Tools
  • Applications
  • Pretraining
  • Fine-tuning
  • Serving
  • Prompt Management
  • Datasets
  • Benchmarks
• Learning Resources for LLMs
  • Applications
    • Agent
  • Modeling
  • Training
  • Fine-tuning
  • Fundamentals
  • Books
  • Newsletters
  • Auto-optimization
• Understanding LLMs
• Social Accounts & Community
  • Social Accounts
  • Community
• Contributing

Libraries & Frameworks & Tools

Applications

Build & Auto-optimize

• AdalFlow - The library to build & auto-optimize LLM applications, from Chatbot, RAG, to Agent by SylphAI.

• dspy - DSPy: The framework for programming—not prompting—foundation models.

Build

• LlamaIndex — A Python library for augmenting LLM apps with data.
• LangChain — A popular Python/JavaScript library for chaining sequences of language model prompts.
• Haystack — Python framework that allows you to build applications powered by LLMs.
• Instill Core — A platform built with Go for orchestrating LLMs to create AI applications.

Prompt Optimization

• AutoPrompt - A framework for prompt tuning using Intent-based Prompt Calibration.
• PromptFify - A library for prompt engineering that simplifies NLP tasks (e.g., NER, classification) using LLMs like GPT.

Others

• LiteLLM - Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format.

Pretraining

• PyTorch - PyTorch is an open source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing.
• TensorFlow - TensorFlow is an open source machine learning library developed by Google.
• JAX - Google’s library for high-performance computing and automatic differentiation.
• tinygrad - A minimalistic deep learning library with a focus on simplicity and educational use, created by George Hotz.
• micrograd - A simple, lightweight autograd engine for educational purposes, created by Andrej Karpathy.

Fine-tuning

• Transformers - Hugging Face Transformers is a popular library for Natural Language Processing (NLP) tasks, including fine-tuning large language models.
• Unsloth - Finetune Llama 3.2, Mistral, Phi-3.5 & Gemma 2-5x faster with 80% less memory!
• LitGPT - 20+ high-performance LLMs with recipes to pretrain, finetune, and deploy at scale.
• AutoTrain - No code fine-tuning of LLMs and other machine learning tasks.

Top Models

• DeepSeek R1 - The most popular GPTo1 comparable open-source reasoning model. Read their technical report + check out their github.

Serving

• TorchServe - An open-source model serving library developed by AWS and Facebook specifically for PyTorch models, enabling scalable deployment, model versioning, and A/B testing.

• TensorFlow Serving - A flexible, high-performance serving system for machine learning models, designed for production environments, and optimized for TensorFlow models but also supports other formats.

• Ray Serve - Part of the Ray ecosystem, Ray Serve is a scalable model-serving library that supports deployment of machine learning models across multiple frameworks, with built-in support for Python-based APIs and model pipelines.

• NVIDIA TensorRT-LLM - TensorRT-LLM is NVIDIA's compiler for transformer-based models (LLMs), providing state-of-the-art optimizations on NVIDIA GPUs.

• NVIDIA Triton Inference Server - A high-performance inference server supporting multiple ML/DL frameworks (TensorFlow, PyTorch, ONNX, TensorRT etc.), optimized for NVIDIA GPU deployments, and ideal for both cloud and on-premises serving.

• ollama - A lightweight, extensible framework for building and running large language models on the local machine.

• llama.cpp - A library for running LLMs in pure C/C++. Supported architectures include (LLaMA, Falcon, Mistral, MoEs, phi and more)

• TGI - HuggingFace's text-generation-inference toolkit for deploying and serving LLMs, built on top of Rust, Python and gRPC.

• vllm - An optimized, high-throughput serving engine for large language models, designed to efficiently handle massive-scale inference with reduced latency.

• sglang - SGLang is a fast serving framework for large language models and vision language models.

• LitServe - LitServe is a lightning-fast serving engine for any AI model of any size. Flexible. Easy. Enterprise-scale.

Prompt Management

• Opik - Opik is an open-source platform for evaluating, testing and monitoring LLM applications
• Agenta - Open-source LLM engineering platform with prompt playground, prompt management, evaluation and observability.

Datasets

Use Cases

• Datasets - A vast collection of ready-to-use datasets for machine learning tasks, including NLP, computer vision, and audio, with tools for easy access, filtering, and preprocessing.
• Argilla - A UI tool for curating and reviewing datasets for LLM evaluation or training.
• distilabel - A library for generating synthetic datasets with LLM APIs or models.

Fine-tuning

• LLMDataHub - A quick guide (especially) for trending instruction finetuning datasets
• LLM Datasets - High-quality datasets, tools, and concepts for LLM fine-tuning.

Pretraining

• IBM LLMs Granite 3.0 - Full list of datasets used to train IBM LLMs Granite 3.0

Benchmarks

• lighteval - A library for evaluating local LLMs on major benchmarks and custom tasks.

• evals - OpenAI's open sourced evaluation framework for LLMs and systems built with LLMs.

• ragas - A library for evaluating and optimizing LLM applications, offering a rich set of eval metrics.

Agent

• TravelPlanner - paper A Benchmark for Real-World Planning with Language Agents.

Learning Resources for LLMs

We will categorize the best resources to learn LLMs, from modeling to training, and applications.

Applications

General

• AdalFlow documentation - Includes tutorials from building RAG, Agent, to LLM evaluation and fine-tuning.

• CS224N - Stanford course covering NLP fundamentals, LLMs, and PyTorch-based model building, led by Chris Manning and Shikhar Murty.

• LLM-driven Data Engineering - A playlist of 6 lectures by Zach Wilson on how LLMs will impact data pipeline development

• LLM Course by Maxime Labonne - An end-to-end course for AI and ML engineers on open source LLMs.

• LLMOps Database by ZenML - 500+ case studies of LLMs and GenAI applications in production, all with summaries and tags for easy browsing.

Agent

Lectures

• OpenAI Pratical Guide to Building Agents- This guide is designed for product and engineering teams exploring how to build their first agents, distilling insights from numerous customer deployments into practical and actionable best practices. It includes frameworks for identifying promising use cases, clear patterns for designing agent logic and orchestration, and best practices to ensure your agents run safely, predictably, and effectively.
• Anthropic's Building Effective Agents - Covers the building blocks and various design patterns for effective agents.
• LLM Agents MOOC - A playlist of 11 lectures by the Berkeley RDI Center on Decentralization & AI, featuring guest speakers like Yuandong Tian, Graham Neubig, Omar Khattab, and others, covering core topics on Large Language Model agents. CS294
• 12 factor agents - Trying to come up with principles for production-grade agents.

Memory

• LLM Agent Memory Survey - A comprehensive survey of memory mechanisms for LLM agents, covering short-term memory, long-term memory, and hybrid approaches.

Optimization

• Awesome-LLM-Agent-Optimization-Papers - A curated list of papers on LLM agent optimization techniques and methodologies.

Projects

• OpenHands - Open source agents for developers by AllHands.
• CAMEL - First LLM multi-agent framework and an open-source community dedicated to finding the scaling law of agents. by CAMEL-AI.
• swarm - Educational framework exploring ergonomic, lightweight multi-agent orchestration. Managed by OpenAI Solution team.
• AutoGen - A programming framework for agentic AI 🤖 by Microsoft.
• CrewAI - 🤖 CrewAI: Cutting-edge framework for orchestrating role-playing, autonomous AI agents.
• TinyTroupe - Simulates customizable personas using GPT-4 for testing, insights, and innovation by Microsoft.

Modeling

• Llama3 from scratch - llama3 implementation one matrix multiplication at a time with PyTorch.
• Interactive LLM visualization - An interactive visualization of transformers. Visualizer
• 3Blue1Brown transformers visualization - 3Blue1Brown's video on how transformers work.
• Self-Attention explained as directed graph - An X post explaining self-attention as a directed graph by Akshay Pachaar.
• LLMs-from-scratch - Github repository with implementation for Build a Large Language Model (From Scratch) by Sebastian Raschka. Covers developing, pretraining, and finetuning GPT-like models from scratch with PyTorch.

Training

• HuggingFace's SmolLM & SmolLM2 training release - HuggingFace's sharing on data curation methods, processed data, training recipes, and all of their code. Github repo.
• Lil'Log - Lilian Weng(OpenAI)'s blog on machine learning, deep learning, and AI, with a focus on LLMs and NLP.
• Chip's Blog - Chip Huyen's blog on training LLMs, including the latest research, tutorials, and best practices.

Fine-tuning

• DPO: Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." Advances in Neural Information Processing Systems 36 (2024). Code.

Fundamentals

• Intro to LLMs - A 1 hour general-audience introduction to Large Language Models by Andrej Karpathy.
• Building GPT-2 from Scratch - A 4 hour deep dive into building GPT2 from scratch by Andrej Karpathy.

Books

• LLM Engineer's Handbook: Master the art of engineering large language models from concept to production by Paul Iusztin , Maxime Labonne. Covers mostly the lifecycle of LLMs, including LLMOps on pipelines, deployment, monitoring, and more. Youtube overview by Paul.
• Build a Large Language Model from Scratch by Sebastian Raschka
• Hands-On Large Language Models: Build, Tune, and Apply LLMs by Jay Alammar , Maarten Grootendorst
• Generative Deep Learning - Teaching machines to Paint, Write, Compose and Play by David Foster
• Large Language Models: A Deep Dive: Bridging Theory and Practice by by Uday Kamath, Kevin Keenan, Garrett Somers, Sarah Sorenson

Newsletters

• Ahead of AI - Sebastian Raschka's Newsletter, covering end-to-end LLMs understanding.
• Decoding ML - Content on building production GenAI, RecSys and MLOps applications.

Auto-optimization

• TextGrad - Automatic ''Differentiation'' via Text -- using large language models to backpropagate textual gradients.

Understanding LLMs

It can be fun and important to understand the capabilities, behaviors, and limitations of LLMs. This can directly help with prompt engineering.

In-context Learning

• Brown, Tom B. "Language models are few-shot learners." arXiv preprint arXiv:2005.14165 (2020).

Reasoning & Planning

• Kambhampati, Subbarao, et al. "LLMs can't plan, but can help planning in LLM-modulo frameworks." arXiv preprint arXiv:2402.01817 (2024).
• Mirzadeh, Iman, et al. "Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models." arXiv preprint arXiv:2410.05229 (2024). By Apple.

Social Accounts & Community

Social Accounts

Social accounts are the best ways to stay up-to-date with the lastest LLM research, industry trends, and best practices.

Community

Contributing

Only with the power of the community can we keep this repo up-to-date and relevant. If you have any suggestions, please open an issue or a direct pull request.

I will keep some pull requests open if I'm not sure if they are not an instant fit for this repo, you could vote for them by adding 👍 to them.

Thanks to the community, this repo is getting read by more people every day.

---

🤝 Please share so we can continue investing in it and make it the go-to resource for LLM engineers—whether they are just starting out or looking to stay updated in the field.

---

If you have any question about this opinionated list, do not hesitate to contact Li Yin