dusty-nv/NanoLLM

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.

382

stars

172

commits

Python

primary language

Oct 18, 2024

updated

dusty-nv.github.io/NanoLLM/

edge-ai

llm-inference

multimodal

rag

speech

vector-database

vision-transformer

README

NanoLLM

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.

[!NOTE]
See dusty-nv.github.io/NanoLLM for docs and Jetson AI Lab for tutorials.

Latest Release: 24.7 (dustynv/nano_llm:24.7-r36.2.0)

Contributors

dusty-nv

168 commits

bigrobinson

2 commits

ms1design

2 commits

Similar repos

Mintplex-Labs/anything-llm

Stop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful…

65,679

shared topics

intel/intel-extension-for-transformers

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for…

2,171

shared topics

dusty-nv/jetson-containers

Machine Learning Containers for NVIDIA Jetson and JetPack-L4T

4,853

same org

genkit-ai/genkit

Open-source framework for building agentic apps in JavaScript, Go, Dart, and Python, built and used…

6,428

shared topics

cactus-compute/cactus

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

5,987

shared topics

SeungjaeLim/K-HyunmooGPT

[K-Data Science Hackaton 3rd Award] Development and use of Korean large-scale generative language…

shared topics

NVIDIA/GenerativeAIExamples

Generative AI reference workflows optimized for accelerated infrastructure and microservice…

4,180

shared topics

yatoooon/TensorRT-LLM

Languages

Python

81.6%

JavaScript

10.1%

HTML

6.8%

CSS

1.5%

dusty-nv/NanoLLM

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.

382

stars

172

commits

Python

primary language

Oct 18, 2024

updated

dusty-nv.github.io/NanoLLM/

edge-ai

llm-inference

multimodal

rag

speech

vector-database

vision-transformer

README

NanoLLM

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.