Bluear7878/FBCache

stars

122

commits

Jupyter Notebook

primary language

Apr 5, 2025

updated

README

Batch	Results	Inference Time (sec)
Batch 1		27.7s
Batch 2		55.0s
Batch 3		1m.24s

##. Potential Overhead from Loop-Based Batch Processing

In some parts of the code, you may see a pattern like:

for (int i = 0; i < batch_size; i++) {
    concat.slice(0, i, i + 1).copy_(encoder_hidden_states.slice(0, i, i + 1));
    qkv_proj.forward(...);
    // ...
}

Contributors

89 commits

23 commits

4 commits

2 commits

Similar repos

nunchux-ai/nunchaku

[ICLR2025 Spotlight] SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

3,947

shared contributors

nunchaku-tech/nunchaku

[ICLR2025 Spotlight] SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

3,947

shared contributors

sayakpaul/diffusers-torchao

End-to-end recipes for optimizing diffusion models with torchao and diffusers (inference and FP8…

399

babdulhakim2/llm-lab

AUGMXNT/inference-benchmarks

Escanord/FAFO

hamelsmu/llama-inference

experiments with inference on llama

103

SqueezeBits/QUICK

QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference

123

Languages

Jupyter Notebook

88.4%

Cuda

5.1%

Python

3.8%

C++

2.3%

README

Batch	Results	Inference Time (sec)
Batch 1		27.7s
Batch 2		55.0s
Batch 3		1m.24s

##. Potential Overhead from Loop-Based Batch Processing

In some parts of the code, you may see a pattern like:

for (int i = 0; i < batch_size; i++) {
    concat.slice(0, i, i + 1).copy_(encoder_hidden_states.slice(0, i, i + 1));
    qkv_proj.forward(...);
    // ...
}