hamelsmu/llama-inference

experiments with inference on llama

103

stars

commits

Python

primary language

Jun 6, 2024

updated

README

llama inference

Exploration of latency on various setups of inference with llama.

Caveats

I didn't explore throughput. That is a deep rabbit hole - I was just exploring latency for a single request. You can tradeoff throughput and latency with various forms of batching requests.
I tried my best to use tools based on the documentation provided.

Contributors

hamelsmu

93 commits

Similar repos

Prathmesh234/simple-inference

Simple inference engine for llama from first principles

ziliangpeng/tinyllm

LLM inference in Python - GPT-2, OPT, LLaMA with KV cache, weight loader, benchmark CLI

eugr/llama-benchy

llama-benchy - llama-bench style benchmarking tool for all backends

697

Leosang-lx/LLM-Inference

Research and develop distributed LLM inference

LoselSpt/LLM

MasterGodzilla/LookAhead

wzy816/llm-pilot

feifeibear/LLMSpeculativeSampling

Fast inference from large lauguage models via speculative decoding

926

Languages

Python

51.3%

Jupyter Notebook

47.2%

Shell

1.5%

hamelsmu/llama-inference

experiments with inference on llama

103

stars

commits

Python

primary language

Jun 6, 2024

updated

README

llama inference

Exploration of latency on various setups of inference with llama.

Caveats

I didn't explore throughput. That is a deep rabbit hole - I was just exploring latency for a single request. You can tradeoff throughput and latency with various forms of batching requests.
I tried my best to use tools based on the documentation provided.

Similar repos

Prathmesh234/simple-inference

Simple inference engine for llama from first principles

ziliangpeng/tinyllm

LLM inference in Python - GPT-2, OPT, LLaMA with KV cache, weight loader, benchmark CLI

eugr/llama-benchy

llama-benchy - llama-bench style benchmarking tool for all backends

697

Leosang-lx/LLM-Inference

Research and develop distributed LLM inference

LoselSpt/LLM

MasterGodzilla/LookAhead

wzy816/llm-pilot

feifeibear/LLMSpeculativeSampling

Fast inference from large lauguage models via speculative decoding

926

Contributors

hamelsmu

93 commits

Languages

Python

51.3%

Jupyter Notebook

47.2%

Shell

1.5%