A survey about LLM inference with RAG on limited resource from the system aspect, etc.
See the codeREADME
Awsome-RAG-LLM-Inference-System
RAG-based LLM inference: a survey about systematic designs
1. GPU Accelerating
- RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving, ISCA 2025
- Hermes: Algorithm-System Co-design for Efficient Retrieval Augmented Generation At Scale, ISCA 2025
- Fast Vector Query Processing for Large Datasets Beyond GPU Memory with Reordered Pipelining, NSDI 2024
- CAGRA: Highly Parallel Graph Construction and Approximate Nearest Neighbor Search for GPUs, ICDE 2024
- Towards High-throughput and Low-latency Billion-scale Vector Search via CPU/GPU Collaborative Filtering and Re-ranking, FAST 2025
- GGNN: Graph-Based GPU Nearest Neighbor Search, TBD 2023
- An Adaptive Vector Index Partitioning Scheme for Low-Latency RAG Pipeline, Arxiv 2025
- BANG: Billion-Scale Approximate Nearest Neighbor Search using a Single GPU, Arxiv 2025 #
- PilotANN: Memory-Bounded GPU Acceleration for Vector Search, Arxiv 2025
- VecFlow: A High-Performance Vector Data Management System for Filtered-Search on GPUs, SIGMOD 2026
- A Systematic Characterization of LLM Inference on GPUs, Arxiv 2025
- A Real-Time Adaptive Multi-Stream GPU System for Online Approximate Nearest Neighborhood Search, CIKM 2024
2. Data Reuse and Pipeline Parallelism
2.1 Data Reuse
- Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation, SIGMOD 2025
- CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion, Eurosys 2025 (Best Paper)
- Leveraging Approximate Caching for Faster Retrieval-Augmented Generation, EuroMLSys 2025
- SPFresh: Incremental In-Place Update for Billion-Scale Vector Search, SOSP 2023
- Revisiting the Index Construction of Proximity Graph-Based Approximate Nearest Neighbor Search, VLDB 2025
- EdgeRAG: Online-Indexed RAG for Edge Devices, Arxiv 2024
- Don't do rag: When cache-augmented generation is all you need for knowledge tasks, WWW 2025
- A Generative Caching System for Large Language Models, Arxiv 2025
- MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation, WWW 2025
- HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse, Arxiv 2025
- CacheFocus: Dynamic Cache Re-Positioning for Efficient Retrieval-Augmented Generation, Arxiv 2025
- METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation, SOSP 2025
- CacheClip: Accelerating RAG with Effective KV Cache Reuse, Arxiv 2025
- CONTEXTPILOT: FAST LONG-CONTEXT INFERENCE VIA CONTEXT REUSE, MLSYS 2026
- RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding, ICML 2025
- Turborag: Accelerating retrieval-augmented generation with precomputed kv caches for chunked text, EMNLP 2025
- OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learning, ICLR 2025 WKSP
- AdaCache: Adaptive Caching and Context Augmentation for Efficient LLM Serving, ICLR 2026
- Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting, ICLR 2025
2.2 Pipeline Parallelism
- PipeRAG: Fast Retrieval-Augmented Generation via Adaptive Pipeline Parallelism, KDD 2025 #
- RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU, Arxiv 2025 #
- Quake: Adaptive Indexing for Vector Search, OSDI 2025
- iQAN: Fast and Accurate Vector Search with Efficient Intra-Query Parallelism on Multi-Core Architectures, PPoPP 2023
- ParlayANN: Scalable and Deterministic Parallel Graph-Based Approximate Nearest Neighbor Search Algorithms, PPoPP 2024
- DGRAG: Distributed Graph-based Retrieval-Augmented Generation in Edge-Cloud Systems, Arxiv 2025
- RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation, Arxiv 2025
- ELERAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval, Arxiv 2025
- Patchwork: A Unified Framework for RAG Serving, Arxiv 2025
- AquaPipe: A Quality-Aware Pipeline for Knowledge Retrieval and Large Language Models, SIGMOD 2025
- VSAG: An Optimized Search Framework for Graph-based Approximate Nearest Neighbor Search, VLDB 2025
- Efficient Graph-Based Approximate Nearest Neighbor Search Achieving: Low Latency Without Throughput Loss, Arxiv 2025
- DepCache: A KV Cache Management Framework for GraphRAG with Dependency Attention, SIGMOD 2026
- VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG, HPCA 2026
- HedraRAG: Co-Optimizing Generation and Retrieval for Heterogeneous RAG Workflows, SOSP 2025
- CoEdge-RAG: Optimizing Hierarchical Scheduling for Retrieval-Augmented LLMs in Collaborative Edge Computing, RTSS 2025
- AdaRAG: Adaptive Optimization for Retrieval Augmented Generation with Multilevel Retrievers at the Edge, INFOCOM 2025
- Patchwork: A Unified Framework for RAG Serving
3. External Memory Accelerating
3.1 Processing in memory
- Scalable Billion-point Approximate Nearest Neighbor Search Using SmartSSDs, ATC 2024
- Accelerating large-scale graph-based nearest neighbor search on a computational storage platform, TC 2022
- Accelerating Retrieval-Augmented Generation, ASPLOS 2025
- NDSEARCH: Accelerating Graph-Traversal-Based Approximate Nearest Neighbor Search through Near Data Processing, ISCA 2024 #
- Fast Graph Vector Search via Hardware Acceleration and Delayed-Synchronization, VLDB 2025 #
- Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models, VLDB 2025 #
- A SmartSSD-based Near Data Processing Architecture for Scalable Billion-point Approximate Nearest Neighbor Search, ToS 2025
- REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing, ISCA 2025
- DReX: Accurate and Scalable Dense Retrieval Acceleration via Algorithmic-Hardware Codesign ISCA 2025
- Robust Implementation of Retrieval-Augmented Generation on Edge-based Computing-in-Memory Architectures, ICCAD 2024
- Compute-Enabled CXL Memory Expansion for Efficient Retrieval Augmented Generation, IEEE Micro 2025
- Performance Analysis and CXL Memory Optimization in Cluster-based RAG Systems,IEEE Transactions on Computers 2025
- HDANNS: In-Memory Hyperdimensional Computing for Billion-Scale Approximate Nearest Neighbour Search Acceleration, TCSAI 2025
- DRIM-ANN: An Approximate Nearest Neighbor Search Engine based on Commercial DRAM-PIMs, Arxiv 2024
- Co-design Hardware and Algorithm for Vector Search, SC 2023
- DF-GAS: a Distributed FPGA-as-a-Service Architecture towards Billion-Scale Graph-based Approximate Nearest Neighbor Search, MICRO 2023
- Accelerating Graph-based Vector Search via Delayed-Synchronization Traversal, VLDB 2025
- In-Storage Acceleration of Retrieval Augmented Generation as a Service, ISCA 2025
- No atomics, no problem. Developing a RAG pipeline for shared CXL memory, TC 2026
3.2 Second-tier Memory
- Achieving Low-Latency Graph-Based Vector Search via Aligning Best-First Search Algorithm with SSD, OSDI 2025
- Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segment, SIGMOD 2024
- Characterizing the Dilemma of Performance and Index Size in Billion-Scale Vector Search and Breaking It with Second-Tier Memory, Arxiv 2024
- HM-ANN: Efficient Billion-Point Nearest Neighbor Search on Heterogeneous Memory, NIPS 2020
- LM-DiskANN: Low Memory Footprint in Disk-Native Dynamic Graph-Based ANN Indexing, BigData 2023
- DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node, NIPS 2019
- LSM-VEC: A Large-Scale Disk-Based System for Dynamic Vector Search, SOSP 2025
- FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search, Arxiv 2021
- Disk-Based Shared KV Cache Management for Fast Inference in Multi-Instance LLM RAG Systems, CLOUD 2025
- CUBO: Self-Contained Retrieval-Augmented Generation on Consumer Laptops 10 GB Corpora, 16 GB RAM, Single-Device Deployment, Arxiv 2026
4. RAG Security
- Machine Against the RAG: Jamming RetrievalAugmented Generation with Blocker Documents, USENIX Security 2025
- THROUGH THE STEALTH LENS: RETHINKINGATTACKS AND DEFENSES IN RAG, Arxiv 2025
- Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation, Arxiv 2026
- Addressing Corpus Knowledge Poisoning Attacks on RAG Using Sparse Attention, Arxiv 2026
- Sector-Aware Cyber Threat Intelligence: A Security-EnhancedRAG Framework for Precision Threat Analysis, Journal of Information Security Research 2026
- Push and Pull : Defending Against Retrieval Poisoning Attacks via Embedding Space Reshaping, ACM Trans. Inf. Syst. 2026
5. Application of RAG
- For Query Optimizer: SEFRQO: A Self-Evolving Fine-Tuned RAG-Based Query Optimizer, SIGMOD 2026
- For Table QA: AixelAsk: A Stepwise-Guided Retrieval and Reasoning Framework for Large Table QA, SIGMOD 2026
- For drug: RAG-based architectures for drug side effectretrieval using compact LLMs, Scientific Reports
llm-inference
rag
system-design
Contributors
ustc-sunny
24 commits
Similar repos
A survey about LLM inference with RAG on limited resource from the system aspect, etc.
See the codeREADME
Awsome-RAG-LLM-Inference-System
RAG-based LLM inference: a survey about systematic designs
1. GPU Accelerating
- RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving, ISCA 2025
- Hermes: Algorithm-System Co-design for Efficient Retrieval Augmented Generation At Scale, ISCA 2025
- Fast Vector Query Processing for Large Datasets Beyond GPU Memory with Reordered Pipelining, NSDI 2024
- CAGRA: Highly Parallel Graph Construction and Approximate Nearest Neighbor Search for GPUs, ICDE 2024
- Towards High-throughput and Low-latency Billion-scale Vector Search via CPU/GPU Collaborative Filtering and Re-ranking, FAST 2025
- GGNN: Graph-Based GPU Nearest Neighbor Search, TBD 2023
- An Adaptive Vector Index Partitioning Scheme for Low-Latency RAG Pipeline, Arxiv 2025
- BANG: Billion-Scale Approximate Nearest Neighbor Search using a Single GPU, Arxiv 2025 #
- PilotANN: Memory-Bounded GPU Acceleration for Vector Search, Arxiv 2025
- VecFlow: A High-Performance Vector Data Management System for Filtered-Search on GPUs, SIGMOD 2026
- A Systematic Characterization of LLM Inference on GPUs, Arxiv 2025
- A Real-Time Adaptive Multi-Stream GPU System for Online Approximate Nearest Neighborhood Search, CIKM 2024
2. Data Reuse and Pipeline Parallelism
2.1 Data Reuse
- Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation, SIGMOD 2025
- CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion, Eurosys 2025 (Best Paper)
- Leveraging Approximate Caching for Faster Retrieval-Augmented Generation, EuroMLSys 2025
- SPFresh: Incremental In-Place Update for Billion-Scale Vector Search, SOSP 2023
- Revisiting the Index Construction of Proximity Graph-Based Approximate Nearest Neighbor Search, VLDB 2025
- EdgeRAG: Online-Indexed RAG for Edge Devices, Arxiv 2024
- Don't do rag: When cache-augmented generation is all you need for knowledge tasks, WWW 2025
- A Generative Caching System for Large Language Models, Arxiv 2025
- MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation, WWW 2025
- HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse, Arxiv 2025
- CacheFocus: Dynamic Cache Re-Positioning for Efficient Retrieval-Augmented Generation, Arxiv 2025
- METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation, SOSP 2025
- CacheClip: Accelerating RAG with Effective KV Cache Reuse, Arxiv 2025
- CONTEXTPILOT: FAST LONG-CONTEXT INFERENCE VIA CONTEXT REUSE, MLSYS 2026
- RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding, ICML 2025
- Turborag: Accelerating retrieval-augmented generation with precomputed kv caches for chunked text, EMNLP 2025
- OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learning, ICLR 2025 WKSP
- AdaCache: Adaptive Caching and Context Augmentation for Efficient LLM Serving, ICLR 2026
- Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting, ICLR 2025
2.2 Pipeline Parallelism
- PipeRAG: Fast Retrieval-Augmented Generation via Adaptive Pipeline Parallelism, KDD 2025 #
- RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU, Arxiv 2025 #
- Quake: Adaptive Indexing for Vector Search, OSDI 2025
- iQAN: Fast and Accurate Vector Search with Efficient Intra-Query Parallelism on Multi-Core Architectures, PPoPP 2023
- ParlayANN: Scalable and Deterministic Parallel Graph-Based Approximate Nearest Neighbor Search Algorithms, PPoPP 2024
- DGRAG: Distributed Graph-based Retrieval-Augmented Generation in Edge-Cloud Systems, Arxiv 2025
- RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation, Arxiv 2025
- ELERAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval, Arxiv 2025
- Patchwork: A Unified Framework for RAG Serving, Arxiv 2025
- AquaPipe: A Quality-Aware Pipeline for Knowledge Retrieval and Large Language Models, SIGMOD 2025
- VSAG: An Optimized Search Framework for Graph-based Approximate Nearest Neighbor Search, VLDB 2025
- Efficient Graph-Based Approximate Nearest Neighbor Search Achieving: Low Latency Without Throughput Loss, Arxiv 2025
- DepCache: A KV Cache Management Framework for GraphRAG with Dependency Attention, SIGMOD 2026
- VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG, HPCA 2026
- HedraRAG: Co-Optimizing Generation and Retrieval for Heterogeneous RAG Workflows, SOSP 2025
- CoEdge-RAG: Optimizing Hierarchical Scheduling for Retrieval-Augmented LLMs in Collaborative Edge Computing, RTSS 2025
- AdaRAG: Adaptive Optimization for Retrieval Augmented Generation with Multilevel Retrievers at the Edge, INFOCOM 2025
- Patchwork: A Unified Framework for RAG Serving
3. External Memory Accelerating
3.1 Processing in memory
- Scalable Billion-point Approximate Nearest Neighbor Search Using SmartSSDs, ATC 2024
- Accelerating large-scale graph-based nearest neighbor search on a computational storage platform, TC 2022
- Accelerating Retrieval-Augmented Generation, ASPLOS 2025
- NDSEARCH: Accelerating Graph-Traversal-Based Approximate Nearest Neighbor Search through Near Data Processing, ISCA 2024 #
- Fast Graph Vector Search via Hardware Acceleration and Delayed-Synchronization, VLDB 2025 #
- Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models, VLDB 2025 #
- A SmartSSD-based Near Data Processing Architecture for Scalable Billion-point Approximate Nearest Neighbor Search, ToS 2025
- REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing, ISCA 2025
- DReX: Accurate and Scalable Dense Retrieval Acceleration via Algorithmic-Hardware Codesign ISCA 2025
- Robust Implementation of Retrieval-Augmented Generation on Edge-based Computing-in-Memory Architectures, ICCAD 2024
- Compute-Enabled CXL Memory Expansion for Efficient Retrieval Augmented Generation, IEEE Micro 2025
- Performance Analysis and CXL Memory Optimization in Cluster-based RAG Systems,IEEE Transactions on Computers 2025
- HDANNS: In-Memory Hyperdimensional Computing for Billion-Scale Approximate Nearest Neighbour Search Acceleration, TCSAI 2025
- DRIM-ANN: An Approximate Nearest Neighbor Search Engine based on Commercial DRAM-PIMs, Arxiv 2024
- Co-design Hardware and Algorithm for Vector Search, SC 2023
- DF-GAS: a Distributed FPGA-as-a-Service Architecture towards Billion-Scale Graph-based Approximate Nearest Neighbor Search, MICRO 2023
- Accelerating Graph-based Vector Search via Delayed-Synchronization Traversal, VLDB 2025
- In-Storage Acceleration of Retrieval Augmented Generation as a Service, ISCA 2025
- No atomics, no problem. Developing a RAG pipeline for shared CXL memory, TC 2026
3.2 Second-tier Memory
- Achieving Low-Latency Graph-Based Vector Search via Aligning Best-First Search Algorithm with SSD, OSDI 2025
- Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segment, SIGMOD 2024
- Characterizing the Dilemma of Performance and Index Size in Billion-Scale Vector Search and Breaking It with Second-Tier Memory, Arxiv 2024
- HM-ANN: Efficient Billion-Point Nearest Neighbor Search on Heterogeneous Memory, NIPS 2020
- LM-DiskANN: Low Memory Footprint in Disk-Native Dynamic Graph-Based ANN Indexing, BigData 2023
- DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node, NIPS 2019
- LSM-VEC: A Large-Scale Disk-Based System for Dynamic Vector Search, SOSP 2025
- FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search, Arxiv 2021
- Disk-Based Shared KV Cache Management for Fast Inference in Multi-Instance LLM RAG Systems, CLOUD 2025
- CUBO: Self-Contained Retrieval-Augmented Generation on Consumer Laptops 10 GB Corpora, 16 GB RAM, Single-Device Deployment, Arxiv 2026
4. RAG Security
- Machine Against the RAG: Jamming RetrievalAugmented Generation with Blocker Documents, USENIX Security 2025
- THROUGH THE STEALTH LENS: RETHINKINGATTACKS AND DEFENSES IN RAG, Arxiv 2025
- Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation, Arxiv 2026
- Addressing Corpus Knowledge Poisoning Attacks on RAG Using Sparse Attention, Arxiv 2026
- Sector-Aware Cyber Threat Intelligence: A Security-EnhancedRAG Framework for Precision Threat Analysis, Journal of Information Security Research 2026
- Push and Pull : Defending Against Retrieval Poisoning Attacks via Embedding Space Reshaping, ACM Trans. Inf. Syst. 2026
5. Application of RAG
- For Query Optimizer: SEFRQO: A Self-Evolving Fine-Tuned RAG-Based Query Optimizer, SIGMOD 2026
- For Table QA: AixelAsk: A Stepwise-Guided Retrieval and Reasoning Framework for Large Table QA, SIGMOD 2026
- For drug: RAG-based architectures for drug side effectretrieval using compact LLMs, Scientific Reports
llm-inference
rag
system-design
Similar repos
Contributors
ustc-sunny
24 commits