NLP-KG Search

NLP-KG Search is a search and question-answering system for NLP/ML papers. The project combines lexical search, semantic search, and a RAG agent to help users retrieve papers, inspect paper details, and ask questions through a web interface.

1. Project Overview

The system includes the following main components:

• BM25: performs lexical keyword search over paper titles and abstracts.
• ChromaDB: stores vector embeddings for semantic search.
• Sentence Transformers: generates embeddings for paper abstracts and full texts.
• FlashRank: reranks semantic search results.
• RRF: combines lexical and semantic results for hybrid search.
• LangGraph: orchestrates the RAG agent workflow.
• Gemini: classifies intent, generates answers, and evaluates answer quality.
• arXiv API: retrieves external papers when the internal corpus is insufficient.
• Streamlit: provides the web interface for search, paper details, and RAG chat.

Important project structure:

.
|-- run_agent.py                 # CLI entry point for the RAG agent
|-- requirements.txt             # Main dependencies
|-- .env.example                 # Environment configuration template
|-- data/
|   |-- data_processed/          # Processed paper data
|   `-- src/                     # Data processing scripts
|-- src/
|   |-- agent/                   # LangGraph agent and RAG nodes
|   |-- bm25/                    # BM25 search
|   |-- chromadb/                # Abstract vector store and semantic search
|   |-- chroma_fulltext/         # Full-text vector store for RAG chat
|   |-- web/                     # Streamlit web app
|   `-- search_engine_for_rag.py # Hybrid search / RRF
`-- md/                          # System flow documentation

2. Environment Setup Instructions

Environment requirements:

• Python 3.10 or later.
• Git.
• Internet access for the first-time download of embedding/reranker models and external API calls.
• A Google Gemini API key if you want to run the RAG agent or chat features.

Create a virtual environment:

python -m venv .venv

Activate it on Windows PowerShell:

.venv\Scripts\Activate.ps1

Activate it on macOS/Linux:

source .venv/bin/activate

Create the environment configuration file:

copy .env.example .env

On macOS/Linux:

cp .env.example .env

Open .env and configure the important variables:

GOOGLE_API_KEY=./Google_api_key.txt
GEMINI_MODEL=gemini-2.5-flash
MAX_CONTEXT_TOKENS=2000
LLM_TEMPERATURE=0.2
LLM_MAX_TOKENS=2000
STREAMLIT_PORT=8501

There are two ways to provide the Gemini key:

• Put the API key directly in the GOOGLE_API_KEY variable.
• Or create Google_api_key.txt in the project root and keep GOOGLE_API_KEY=./Google_api_key.txt.

If you run the data pipeline with Semantic Scholar, create this file:

data/src/api.txt

Then place the Semantic Scholar API key inside that file.

3. Dependency Installation Steps

Install all required dependencies:

pip install -r requirements.txt

4. How To Train The Model

This project does not train a supervised neural model from scratch. In this project, "training" means processing the paper data, generating embeddings, and rebuilding the search indexes.

4.1. Process Paper Data

Run the scripts in data/src/ in order if you need to rebuild the dataset:

python data/src/1_loose_filter.py
python data/src/2_map.py
python data/src/3_strict_filter.py
python data/src/4_clean_and_merge.py
python data/src/5_check_is_survey.py

After the pipeline finishes, the cleaned data is stored in data/data_processed/.

4.2. Build The Abstract Vector Store

This vector store is used for semantic search over paper titles and abstracts:

python src/chromadb/ingest.py

Default output:

src/chromadb/chroma_store_abstracts/

4.3. Build The Full-Text Vector Store

This vector store is used for RAG chat over full-text papers:

python src/chroma_fulltext/ingest.py

Default output:

data/chroma_store_fulltext/

4.4. Run The Search Pipeline

python src/search_engine_for_rag.py

This pipeline combines BM25, ChromaDB, and RRF to produce hybrid search results.

5. How To Run Inference Or The Deployed System

5.1. Run CLI Inference

Run a single query:

python run_agent.py --query "What is BERT pretraining?"

Run interactive question answering:

python run_agent.py --interactive

Run built-in examples:

python run_agent.py --examples

Run batch queries from a JSON file:

python run_agent.py --batch queries.json --output results.json

5.2. Run The Streamlit Web App

streamlit run src/web/app.py

After the app starts, open:

http://localhost:8501

The web app supports:

• Home page: enter a query and enable/disable semantic search.
• Results page: view, filter, and rank retrieved papers.
• Detail page: inspect detailed information for a selected paper.
• Chat page: ask questions using the RAG agent.

If the app reports that ChromaDB cannot be found, rebuild the indexes:

python src/chromadb/ingest.py
python src/chroma_fulltext/ingest.py

5.3. Run With Docker

Docker is the easiest way to run the deployed Streamlit application. A ready-built image is available on Docker Hub:

nguyennhattk19/nlp-kg-search:latest

Pull the image:

docker pull nguyennhattk19/nlp-kg-search:latest

Run the container:

docker run -p 8501:8501 -e GOOGLE_API_KEY="your_google_api_key" nguyennhattk19/nlp-kg-search:latest

Replace your_google_api_key with your real Google Gemini API key.

After the container starts, open:

http://localhost:8501

To stop the app, press Ctrl+C in the terminal running the container.

If you want to run the container in the background:

docker run -d --name nlp-kg-search -p 8501:8501 -e GOOGLE_API_KEY="your_google_api_key" nguyennhattk19/nlp-kg-search:latest

Then stop and remove it with:

docker stop nlp-kg-search
docker rm nlp-kg-search

If you want to build the image locally instead of pulling it from Docker Hub:

docker compose build

Then run the local image:

docker run -p 8501:8501 -e GOOGLE_API_KEY="your_google_api_key" nlp-kg-search:latest

Notes:

• Do not bake .env, Google_api_key.txt, or API tokens into the image.
• Pass secrets at runtime with -e GOOGLE_API_KEY="...".
• The image can be large because the ChromaDB stores are a few GB.

6. Description Of Deployment Method

The deployment method is a Streamlit application running in a Python environment with the required data files and vector stores available locally.

Deployment steps:

1. Prepare a server or local machine with Python 3.10+.
2. Clone the project source code.
3. Create a virtual environment and install dependencies.
4. Configure .env and the required API keys.
5. Make sure the processed data and ChromaDB stores are available, or rebuild them with the ingest scripts.
6. Start the application with Streamlit.

Local deployment command:

streamlit run src/web/app.py 

Deployment architecture:

User Browser
    |
    v
Streamlit Web App
    |
    |-- BM25 lexical search
    |-- ChromaDB semantic search
    |-- FlashRank reranker
    |-- LangGraph RAG agent
    |-- Gemini API
    `-- arXiv API fallback

The application does not require a separate model server. Embedding and reranker models are loaded directly inside the Python process, ChromaDB is stored locally on disk, and the LLM is accessed through the Gemini API.