yuliu625/Yu-RAG-Toolkit

Modular RAG framework built with LangChain & LangGraph: supporting multi-modal embeddings, semantic chunking, and advanced agentic retrieval strategies.

Python

372 commits

updated Aug 16, 2026

See the code

README

🚀 RAG Toolkit

This is a toolkit specifically designed for RAG (Retrieval-Augmented Generation) research and engineering. The project systematically deconstructs the entire pipeline—from raw data processing to complex state-machine retrieval—aiming to provide a pluggable and extensible foundation for experimentation.

📌 Core Architecture

The repository is modularized according to the RAG lifecycle, ensuring that the logic of each stage is highly decoupled:

Stage	Module Name	Core Function & Positioning
Data Ingestion	`langchain_loading`	Encapsulates standardized loading methods; supports various unstructured data sources.
Semantic Splitting	`langchain_splitting`	Diverse document chunking strategies to optimize semantic integrity.
Representation	`langchain_embedding`	Integration of mainstream Embedding models and multi-modal vectorization schemes.
Vector Storage	`langchain_storing`	Vector index construction, metadata management, and efficient persistence.
Perception Processing	`langchain_document_processing`	`Document` object transformation logic, bridging retrieval and generation.
Traditional Retrieval	`langchain_retrievers`	[Legacy] Implementation of basic LangChain native retrievers.
Advanced Retrieval	`langgraph_retrievers`	[Active] Complex RAG strategies implemented via LangGraph state machines.

🛠️ Module Details

1. Pipeline Foundations

langchain_loading & langchain_splitting: Solves standardized preprocessing from multi-source data (PDF, Markdown, Web) into semantic chunks.
langchain_document_processing: Specifically handles Document objects. This serves as the preprocessing layer after retrieval but before inputting into LLMs & VLMs.

2. Representation & Storage

langchain_embedding:
- Covers integration schemes for mainstream text embedding models.
- Highlight: Implements a multi-modal embedding solution based on ChromaDB, supporting cross-modal vector retrieval experiments.
langchain_storing: Focuses on the engineering implementation of vector databases, including index optimization, persistent storage, and metadata filtering.

3. Evolution of Retrieval Strategies

This section records the evolution of retrieval paradigms from static to dynamic:

langchain_retrievers (Basics):
- Archives native retrieval techniques such as Multi-query and Contextual Compression.
- Note: This module is no longer updated and is kept for research comparison only.
langgraph_retrievers (Advanced):
- Core Development Module. Utilizes LangGraph to upgrade retrieval logic from simple Chains to complex state machines.
- Note: This module is under active development.

This repository is part of my personal research ecosystem. You can use it in conjunction with the following:

RAG-Toolkit: Core retrieval-augmented toolset.
Agent-Development-Toolkit: Focused on building logic for Intelligent Agents.
Deep-Learning-Toolkit: A general-purpose foundation for deep learning tasks.
Data-Science-Toolkit: Tools for data science and preprocessing.

information-retrieval

langchain

langgraph

llm

multi-modal

rag

vector-database

Similar repos

Danielskry/Awesome-RAG

😎 Awesome list of Retrieval-Augmented Generation (RAG) applications in Generative AI.

1,400

jinbo0906/Awesome_RAG

awesome_rag：模块化 RAG 系统构建与前沿探索

lizhe2004/Awesome-LLM-RAG-Application

the resources about the application based on LLM with RAG pattern

1,662

datawhalechina/all-in-rag

🔍大模型应用开发实战一：RAG 技术全栈指南，在线阅读地址：https://datawhalechina.github.io/all-in-rag/

12,265

KalyanKS-NLP/rag-zero-to-hero-guide

Comprehensive guide to learn RAG from basics to advanced.

1,458

Poll-The-People/awesome-rag

awesome-rag: a collection of awesome thing related to Retrieval-Augmented Generation

188

vivy-yi/rag-tutorial

完整的RAG技术教程 - 从基础概念到生产部署，系统化掌握检索增强生成技术。包含4个模块、20章内容、17个Jupyter Notebooks、6个企业级实战案例

102

HKUDS/RAG-Anything

"RAG-Anything: All-in-One RAG Framework"

23,793