PDF & EPUB Text Extractor

A powerful web application for bulk extraction and preprocessing of text from PDF and EPUB files, optimized for technical content and machine learning dataset preparation.

✨ Features

• 🔒 Fully Offline: Works completely offline with LM Studio - no internet required
• Bulk Processing: Process multiple PDF and EPUB files simultaneously
• Smart OCR: Vision-based OCR for scanned documents with multiple quality modes
• Technical Content: Preserves code blocks, mathematical formulas, and technical notation
• Text Preprocessing: Advanced preprocessing for ML datasets (stop words, lemmatization, POS tagging)
• Sentiment Analysis: LLM-powered sentiment analysis
• ML Export Formats: PyTorch, TensorFlow, Hugging Face, CSV formats with train/val/test splits
• Local-First LLM: Designed for LM Studio, also works with Ollama or any OpenAI-compatible API
• Privacy-Focused: All processing happens locally, your data never leaves your machine

🚀 Quick Start

Run Locally (Development)

This project is a React + Vite web app. To run it locally:

# Install dependencies
npm install

# Start the dev server
npm run dev

Then open the URL printed in the terminal (usually http://localhost:5173).

Optional Local API (Debugging)

If you want a Node-based API for testing PDF/EPUB extraction without a browser, run:

npm run local-api

The API listens on http://localhost:8787/api/process and expects JSON:

{
  "fileName": "example.pdf",
  "fileType": "pdf",
  "contentBase64": "<base64-encoded-file>"
}

Python Requirements (Optional)

The app exports Python dataset loader templates (e.g., dataset_loader.py and tf_dataset_loader.py). If you plan to use those templates, install the optional dependencies from:

pip install -r requirements.txt

1. Install LM Studio (Default & Recommended)

LM Studio provides the best offline experience with a simple GUI:

1. Download LM Studio from https://lmstudio.ai
2. Install and open LM Studio
3. Download a vision model (for OCR support):
   • Go to "Discover" tab
   • Search for "llava" or "llama3.2-vision"
   • Download a model (recommended: llava-v1.6-7b or llama3.2-vision)
4. Start the Local Server:
   • Click "Local Server" tab
   • Load your downloaded model
   • Click "Start Server" (default port: 1234)

Alternative: Using Ollama (command-line):

# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Pull a vision model for OCR support
ollama pull llama3.2-vision

# Ollama starts automatically, or run:
ollama serve

2. Configure the App

1. Open the application (npm run dev)
2. Click "LLM Settings"
3. Default is already "LM Studio (Local) - Recommended"
4. Verify settings match your LM Studio server:
   • API URL: http://localhost:1234/v1
   • Model: Match the model you loaded in LM Studio
   • API Key: Leave blank
5. Click "Save Settings"

✅ That's it! You can now work completely offline.

3. Start Processing

1. Drag and drop PDF/EPUB files
2. Configure OCR and preprocessing options
3. Click "Process Files"
4. Export in your preferred format

📚 Documentation

• 📑 Documentation Index - Complete documentation navigation
• ⚡ Quick Start Guide - Get running in 5 minutes
• 🧪 Testing LM Studio Integration - Verify offline functionality works
• 📖 Full Documentation - Complete feature guide and troubleshooting
• 🔧 LLM Configuration - Detailed guide for configuring different LLM providers
• 💻 API Reference - Technical documentation for developers
• 📋 Product Requirements - Detailed specifications and design decisions
• 🤖 Personal AI Training Guide - Build your own AI with LoRA + RAG

🔧 LLM Configuration

The app is designed for offline-first operation with LM Studio:

Recommended Models

For OCR (vision required):

• llama3.2-vision (Ollama) - Best quality
• llava (Ollama) - Good balance
• gpt-4o (OpenAI) - Fast, cloud-based

For text tasks only:

• llama3.2 (Ollama)
• mistral (Ollama)
• gpt-4o-mini (OpenAI)

🎯 Use Cases

Technical Document Processing

• Extract text from programming books
• Preserve code examples and syntax
• Maintain mathematical notation
• Keep technical terminology intact

ML Dataset Creation

• Prepare training data for LLM fine-tuning
• Create LoRA datasets
• Generate train/validation/test splits
• Export in ML framework formats

Content Analysis

• Sentiment analysis of documents
• Statistical analysis (word counts, etc.)
• Batch processing for large document sets

🤖 Personal AI Training Starter Kit

NEW! Build your own personal AI that grows with you over time.

This repository now includes a complete starter kit for training a domain-specific AI model on your personal knowledge:

• 📓 Jupyter Notebooks: Step-by-step guides for LoRA fine-tuning and RAG knowledge bases
• 📝 Data Templates: Ready-to-use templates for capturing personal knowledge
• 🧠 LoRA Training: Fine-tune Mistral 7B on your writing style and thinking patterns
• 🔍 RAG Integration: Build a local vector database with your documents
• 🖥️ Optimized for RTX 5080: Memory-efficient training with 4-bit quantization
• 🔒 Privacy-First: Everything runs locally, no cloud dependencies

Quick Start with Personal AI

# Install training dependencies
pip install -r requirements-training.txt

# Start Jupyter
jupyter notebook notebooks/personal_model_training.ipynb

Learn more: Read the comprehensive Personal AI Training Guide

The workflow:

1. Extract text from PDFs/EPUBs using this tool
2. Export in ML formats (PyTorch, HuggingFace, JSON)
3. Combine with personal data templates
4. Train your own LoRA adapter
5. Build a RAG knowledge base
6. Chat with an AI that thinks like you!

🛠️ Technical Stack

• Frontend: React + TypeScript
• Styling: Tailwind CSS + shadcn/ui
• PDF Processing: PDF.js
• EPUB Processing: JSZip
• LLM Integration: OpenAI-compatible API
• Data Persistence: Spark KV store (browser-local)

🔒 Privacy & Security

• ✅ Fully Offline by Default: Works without internet using LM Studio
• ✅ Zero Cloud Dependencies: All processing happens on your machine
• ✅ Your Data Stays Local: Files never leave your computer
• ✅ No Telemetry: No tracking, no analytics, no data collection
• ✅ Browser-Local Storage: Configuration saved in your browser only
• ✅ Client-Side Processing: PDF/EPUB extraction happens in your browser
• ✅ Optional Cloud APIs: Only if you explicitly configure OpenAI or other cloud providers

📦 Export Formats

• Plain Text (.txt)
• JSON Array
• PyTorch Dataset (train/val/test splits)
• TensorFlow Dataset (train/val/test splits)
• Hugging Face Format
• CSV with metadata

🐛 Troubleshooting

Can't connect to LLM?

• Verify Ollama is running: ollama list
• Check API URL includes /v1 path
• Try 127.0.0.1 instead of localhost

OCR not working?

• Ensure you're using a vision model
• Text-only models cannot perform OCR
• Try ollama pull llama3.2-vision

Processing slow?

• Local LLMs are slower but free and private
• Use smaller models for speed
• Consider cloud APIs for faster results

See DOCUMENTATION.md for complete troubleshooting guide.

📝 License

See LICENSE file for details.

🤝 Contributing

Issues and feature requests are welcome! Please check existing documentation before reporting.

---

Note: This application requires a vision-capable LLM for OCR features. Text-only features (sentiment analysis, preprocessing) work with any LLM model.