awesome-llm-utilities

Tooling and frameworks to support building tools that utilize LLM agents

Personal Favorites

• https://github.com/microsoft/JARVIS - JARVIS, a system to connect LLMs with ML community. Paper: https://arxiv.org/pdf/2303.17580.pdf (HuggingGPT)
• https://github.com/hwchase17/langchain - ⚡ Building applications with LLMs through composability ⚡
• https://github.com/jerryjliu/llama_index - LlamaIndex (GPT Index) is a project that provides a central interface to connect your LLM's with external data.
• https://github.com/chatarena/chatarena - Multi-Agent Language Game Environments for LLMs
• https://github.com/irgolic/AutoPR - Fix issues with AI-generated pull requests, powered by ChatGPT. discord.gg/ykk7Znt3K6
• https://github.com/PrefectHQ/marvin - 🤖🪄 A batteries-included library for building AI-powered software. askmarvin.ai
• https://github.com/ShreyaR/guardrails - Adding guardrails to large language models. discord.gg/Jsey3mX98B
• https://github.com/eth-sri/lmql - A query language for programming (large) language models. - lmql.ai
• https://github.com/deepset-ai/haystack - Haystack is an end-to-end NLP framework that enables you to build NLP applications powered by LLMs, Transformer models, vector search and more. Whether you want to perform question answering, answer generation, semantic document search, or build tools that are capable of complex decision making and query resolution, you can use the state-of-the-art NLP models with Haystack to build end-to-end NLP applications solving your use case.
• https://github.com/togethercomputer/OpenChatKit - OpenChatKit provides a powerful, open-source base to create both specialized and general purpose chatbots for various applications. The kit includes an instruction-tuned language models, a moderation model, and an extensible retrieval system for including up-to-date responses from custom repositories. OpenChatKit models were trained on the OIG-43M training dataset, which was a collaboration between Together, LAION, and Ontocord.ai.

---

Agent orchestration, multi-step planning

• https://github.com/microsoft/JARVIS - JARVIS, a system to connect LLMs with ML community. Paper: https://arxiv.org/pdf/2303.17580.pdf (HuggingGPT)
• https://github.com/hwchase17/langchain - ⚡ Building applications with LLMs through composability ⚡
• https://github.com/jerryjliu/llama_index - LlamaIndex (GPT Index) is a project that provides a central interface to connect your LLM's with external data.
• https://github.com/emptycrown/llama-hub - A library of data loaders for LLMs made by the community -- to be used with GPT Index and/or LangChain
• https://github.com/stanfordnlp/dsp - 𝗗𝗦𝗣: Demonstrate-Search-Predict. A framework for composing retrieval and language models for knowledge-intensive NLP.
• https://github.com/amazon-science/mm-cot - Multimodal Chain-of-Thought Reasoning in Language Models
• https://github.com/microsoft/visual-chatgpt - Visual ChatGPT connects ChatGPT and a series of Visual Foundation Models to enable sending and receiving images during chatting. See our paper: Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
• https://github.com/lightaime/camel - CAMEL: Communicative Agents for “Mind” Exploration of Large Scale Language Model Society - www.camel-ai.org
• https://github.com/chatarena/chatarena - Multi-Agent Language Game Environments for LLMs
• https://github.com/google-research/cascades - Python library which enables complex compositions of language models such as scratchpads, chain of thought, tool use, selection-inference, and more.
• https://github.com/mpaepper/llm_agents

Coding agents, Implicitly defined tools, Tools for Developers

• https://github.com/irgolic/AutoPR - Fix issues with AI-generated pull requests, powered by ChatGPT. discord.gg/ykk7Znt3K6
• https://github.com/e2b-dev/e2b - e2b (english2bits) is an IDE powered by AI agents. Developers describe what they want to build by writing documentation. Then let AI agents with access to tools do the coding work. e2b.dev
• https://github.com/dmarx/the-rest-of-the-fucking-owl - Trigger an LLM in your CI/CD to auto-complete your work
• https://github.com/BlackHC/llm-strategy - Directly Connecting Python to LLMs - Dataclasses & Interfaces <-> LLMs blackhc.github.io/llm-strategy
• https://github.com/biobootloader/wolverine - Give your python scripts regenerative healing abilities! Run your scripts with Wolverine and when they crash, GPT-4 edits them and explains what went wrong. Even if you have many bugs it will repeatedly rerun until it's fixed.
• https://github.com/Quansight/pseudocode - pseudocode is a python module which allows users to describe the functions they want via type annotations and docstrings without writing the actual code. It empowers users to iterate with large language models to generate functions which satisfy the user along with defined tests.
• https://github.com/fjrdomingues/autopilot - Dev Autopilot, a tool that uses GPT to read a codebase, create context and solve tasks.
• https://github.com/PrefectHQ/marvin - 🤖🪄 A batteries-included library for building AI-powered software. askmarvin.ai
• https://github.com/microsoft/semantic-kernel

Prompt Construction, Generation Validation and Guidance

• https://github.com/ShreyaR/guardrails - Adding guardrails to large language models. discord.gg/Jsey3mX98B
• https://github.com/benlipkin/probsem - a framework to leverage large language models (LLMs) to assign context-conditional probability distributions over queried strings, with default support for all OpenAI engines and HuggingFace CausalLM models.
• https://github.com/HazyResearch/manifest - How to make prompt programming with Foundation Models a little easier.
• https://github.com/eth-sri/lmql - A query language for programming (large) language models. - lmql.ai
• https://github.com/bigscience-workshop/promptsource - Toolkit for creating, sharing and using natural language prompts.
• https://github.com/newhouseb/clownfish - Constrained Decoding for LLMs against JSON Schema
• https://github.com/kyang6/llmparser - [TypeScript] Classify and extract structured data with LLMs. llmparser.com

IDE extensions, frontends, browser plugins

• https://github.com/MateusZitelli/PromptMate - PromptMate is your AI partner to develop code. 💻 It connects to OpenAI Conversation API (GPT-4, GPT-3.5) and helps you create proper prompts and engage in conversations in VS Code, keeping your code in context. You can easily get code snippets, ask questions, and receive guidance from your AI assistant. Install PromptMate for VS Code.
• https://github.com/cfortuner/plugins - AI plugins for apps like chatGPT :)
• https://github.com/transmissions11/flux
• • https://github.com/socketteer/loom - Multiversal tree writing interface for human-AI collaboration

Annotation, Conversational Information Retrieval, misc

• https://github.com/explosion/prodigy-openai-recipes - This repository contains example code on how to combine zero- and few-shot learning with a small annotation effort to obtain a high-quality dataset with maximum efficiency. Specifically, we use large language models available from OpenAI to provide us with an initial set of predictions, then spin up a Prodigy instance on our local machine to go through these predictions and curate them. This allows us to obtain a gold-standard dataset pretty quickly, and train a smaller, supervised model that fits our exact needs and use-case.
• https://github.com/hwchase17/chat-langchain - This repo is an implementation of a locally hosted chatbot specifically focused on question answering over the LangChain documentation.
• https://github.com/whitead/paper-qa - LLM Chain for answering questions from documents with citations
• https://github.com/jagilley/fact-checker - Fact-checking LLM outputs with langchain
• https://github.com/argilla-io/argilla - ✨ Argilla: Open-source platform empowering teams to build better language models through human feedback. docs.argilla.io
• https://github.com/deepset-ai/haystack - 🔍 Haystack is an open source NLP framework to interact with your data using Transformer models and LLMs (GPT-4, ChatGPT and alike). Haystack offers production-ready tools to quickly build complex decision making, question answering, semantic search, text generation applications, and more. haystack.deepset.ai
• https://github.com/hwchase17/notion-qa

Verctor stores

• https://github.com/chroma-core/chroma
• https://www.pinecone.io/

Generation SDKs

• https://github.com/hyperonym/basaran - Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models.
• https://github.com/NVIDIA/NeMo - NVIDIA NeMo is a conversational AI toolkit built for researchers working on automatic speech recognition (ASR), text-to-speech synthesis (TTS), large language models (LLMs), and natural language processing (NLP). The primary objective of NeMo is to help researchers from industry and academia to reuse prior work (code and pretrained models) and make it easier to create new conversational AI models.
• https://github.com/togethercomputer/OpenChatKit - OpenChatKit provides a powerful, open-source base to create both specialized and general purpose chatbots for various applications. The kit includes an instruction-tuned language models, a moderation model, and an extensible retrieval system for including up-to-date responses from custom repositories. OpenChatKit models were trained on the OIG-43M training dataset, which was a collaboration between Together, LAION, and Ontocord.ai.
• https://github.com/neuml/txtai - 💡 Semantic search and workflows powered by language models. neuml.github.io/txtai
• https://github.com/RasaHQ/rasa - Rasa is an open source machine learning framework to automate text and voice-based conversations. With Rasa, you can build contextual assistants on: . Rasa helps you build contextual assistants capable of having layered conversations with lots of back-and-forth. In order for a human to have a meaningful exchange with a contextual assistant, the assistant needs to be able to use context to build on things that were previously discussed – Rasa enables you to build assistants that can do this in a scalable way.

Autonomy Experiments

• https://github.com/Torantulino/Auto-GPT - An experimental open-source attempt to make GPT-4 fully autonomous.
• https://github.com/yoheinakajima/babyagi
• https://github.com/fullthom/chat-gpt-quine

Misc Research

• https://github.com/posgnu/rci-agent - A codebase for "Language Models can Solve Computer Tasks" - posgnu.github.io/rci-web/
• https://github.com/yizhongw/self-instruct - Aligning pretrained language models with instruction data generated by themselves. https://arxiv.org/abs/2212.10560
• https://github.com/noahshinn024/reflexion - Code for the approach proposed in Reflexion: an autonomous agent with dynamic memory and self-reflection
• https://github.com/GammaTauAI/reflexion-human-eval - This is a spin-off project inspired by the paper: Reflexion: an autonomous agent with dynamic memory and self-reflection. Noah Shinn, Beck Labash, Ashwin Gopinath. Preprint, 2023
• https://github.com/ysymyth/ReAct - GPT-3 prompting code for ICLR 2023 paper ReAct: Synergizing Reasoning and Acting in Language Models.
• https://github.com/GammaTauAI/opentau
• https://github.com/madaan/self-refine - LLMs can generate feedback on their work, use it to improve the output, and repeat this process iteratively.
• https://github.com/Muennighoff/sgpt - SGPT: GPT Sentence Embeddings for Semantic Search
• https://github.com/HKUNLP/instructor-embedding - This repository contains the code and pre-trained models for our paper One Embedder, Any Task: Instruction-Finetuned Text Embeddings. Please refer to our project page for a quick project overview. We introduce Instructor👨‍🏫, an instruction-finetuned text embedding model that can generate text embeddings tailored to any task (e.g., classification, retrieval, clustering, text evaluation, etc.) and domains (e.g., science, finance, etc.) by simply providing the task instruction, without any finetuning. Instructor👨‍ achieves sota on 70 diverse embedding tasks!
• https://github.com/reasoning-machines/pal - PaL: Program-Aided Language Models. reasonwithpal.com
• https://github.com/conceptofmind/toolformer - Open-source implementation of Toolformer: Language Models Can Teach Themselves to Use Tools by Meta AI.
• https://github.com/project-baize/baize-chatbot - Baize is an open-source chat model trained with LoRA. It uses 100k dialogs generated by letting ChatGPT chat with itself. We also use Alpaca's data to improve its performance. We have released 7B, 13B and 30B models. Please refer to the paper for more details.
• https://github.com/CarperAI/OpenELM - This repository is a replication of Evolution Through Large Models, a recent paper from OpenAI exploring the links between large language models (LLMs) and evolutionary computing, particularly focused on code generation.
• https://github.com/IBM/model-recycling - To see the best model to start from, or study the differences between finetuned models, please visit the website. Or read the paper.
• https://github.com/HazyResearch/ama_prompting - This repository contains code for the Ask Me Anything (AMA) prompt-aggregation strategy. The end-to-end AMA approach includes (1) recursively using the language model to transform the task format and prompt and (2) aggregating the predictions of multiple prompts using weak supervision.
• https://github.com/microsoft/LMOps
• https://github.com/AGI-Edgerunners/LLM-Adapters

LLM Red Teaming and Prompt Bypass

• https://github.com/greshake/llm-security - New ways of breaking app-integrated LLMs. We demonstrate potentially brutal consequences of giving LLMs like ChatGPT interfaces to other applications. We propose newly enabled attack vectors and techniques and provide demonstrations of each in this repository. See also: https://greshake.github.io/
• https://github.com/traghav/auto-redteam

Related lists

• https://github.com/HazyResearch/data-centric-ai
• https://github.com/FreedomIntelligence/ReasoningNLP
• https://github.com/Hannibal046/Awesome-LLM
• https://github.com/GT-RIPL/Awesome-LLM-Robotics
• https://github.com/RUCAIBox/LLMSurvey