🤖 Regul𝔸𝕀tions

Banking Compliance Made Intelligent: Navigating regulatory complexity with RAG-enhanced Fine-Tuned LLMs.

🎯 Project Motivation

Banking compliance is a major bottleneck due to dense, constantly changing regulations scattered across multiple sources. RegulAItions centralizes these documents in one searchable place and uses modern LLMs to provide instant, structured, and verifiable regulatory insights. ⚖️

🧩 Problem Statement

General-purpose LLMs are inadequate for banking compliance due to three fundamental failures:

• Hallucinations: Models behave like stochastic parrots, fabricating rules when evidence is missing. 🦜

• Domain Gap: They lack fluency in Israeli banking terminology and regulatory semantics. 🏦

• Unstructured Output: They default to conversational prose instead of machine-consumable JSON required by compliance systems. 📄➡️🧾

🗄️ Data & Data Engineering

• 📁 Data Used:

  • Source Material: 6 official Bank of Israel PDF circulars.

  • Vector Database (RAG): 518 optimized text chunks stored in ChromaDB as high-dimensional vector embeddings. 🔍

• 🔧 Data Augmentation & Generation:

  • Fine-Tuning Gold Set: 2,000+ generated structured Queries, each sample contains query, response, verdict, and exact document location.

  • Hallucination Training: 15–20% Negative Samples (Hard & Soft). These force the model to output "N.A" when evidence is absent, reducing hallucinations and overconfidence. 🛡️

🛠️ Technical Stack & Models

• 🤖 Generative Models (LLMs):

  • Llama 3.1 — strong generalist baseline.
  • Saul-7B-Instruct-v1 — 7B parameter model specialized for legal text. ⚖️

• 🔎 Embedding Models (Encoders):

  • all-MiniLM-L6-v2 — general semantic retrieval.
  • LegalBERT — domain-adapted legal embeddings. 📚

• ⚙️ Training Setup

  • Training Framework: Unsloth for 4-bit quantized, memory-efficient training. ⚡
  • Methodology: LoRA (Low-Rank Adaptation) for Parameter-Efficient Fine-Tuning (PEFT).🎯

🔗 The Hybrid Architecture (FT + RAG)

• 📖 RAG (The “What”)

  • Dynamically retrieves the most relevant regulatory context from the 518-chunk corpus.
  • Provides the model with an “open-book” reference during inference. 📑

• 🧠 Fine-Tuning (The “How”)

  • Teaches the model to reason like a compliance officer.
  • Enforces mastery of a strict JSON schema and legal decision logic.

• 🔀Integration

  • Evaluated four pipelines: (Llama / SaulLM) × (all-MiniLM / LegalBERT).

📥📤 Input / Output Specifications

• Input:

  • Natural language regulatory query. 🗣️

• Output (Strict JSON):

  {
  "verdict": "Yes | No | N.A",
  "quote": "citation anwser is based on",
  "source": "Circular Name, Page Number",
  "explanation": "grounded text reasoning"
  }

🧾 Visual Abstract

📊 Evaluation Metrics & Results

• 📏 Metrics:

  • HitRate@K: Probability that the correct context appears in top-4 retrieved chunks.
  • Verdict Accuracy: Correct Yes/No/N.A classification.
  • Hallucination Rate: Proportion of responses that contain unsupported, fabricated, or incorrectly inferred information not grounded in the retrieved regulatory text.

• 📈 Results:

  • HitRate@K = 86–90%.
  • Verdict Accuracy = 94%-98%.
  • Hallucination Rate = 0%.
  • JSON Integrity: 100% valid JSON post fine-tuning.

Key Finding: SaulLM + all-MiniLM achieved the highest precision on complex legal reasoning tasks. 🏆

📁 Repository Structure

• 📁Presentations – Proposal, interim, and final presentations

• 📁Environment_dependencies - Libraries and environment settings

• 📁Code – Implementation

  • 📁Baseline_notebooks – Notebooks for initial model experiments
  • 📁data_generation_&_validation - Scripts for generating and validating datasets
  • 📁FineTuning - Scripts for Fine-Tuning and inference
    • 📄inference.py - Executes model predictions on input data
    • 📄report_from_inference.py - Analyzes and summerizes inference results
    • 📄train.py - Manages the model training process
  • 📁RAG - Implement RAG pipeline scripts
    • 📁interactive_scripts - Scripts for real-time model interaction
      • 📄interactive_QA.py - Main interactive script
    • 📄run_hybrid_eval.py - Executes full RAG pipeline evaluation
    • 📄build_all_db.py - Builds the specialized allMiniLM-L6-v2 vector DB
    • 📄build_legal_db.py - Builds the specialized LegalBERT vector DB
    • 📄[email protected] - Calculates the retreival accuracy
  • 📄EDA.py – Exploratory data analysis script
  • 📄alpaca_format_and_splitting.py – Converts the raw dataset into alpaca format and splits into Train/Test JSON files for LLM fine-tuning.

• 📁Data – Datasets

  • 📁Regulatory_Rules – Official regulatory documents
  • 📁FT_datasets – Train and Test data sets for Fine Tuning
  • 📁Chunks – Contains chunks of regulation rules in json format
  • 📁RAG_db_MiniLM – MiniLM vector DB storage
  • 📁RAG_db_legal – LegalBERT vector DB storage
  • 📁final_eval_questions – Evaluation datasets for RAG testing
  • 📁raw_datasets – Merge an unformatted source datasets

• 📁Models - Saved model weights and configurations

  • 📁Baseline_LoRA - Pretrained LoRA model checkpoints
  • 📁saul_adapter - Fine-Tuned weights for SaulLM model
  • 📁Llama3.1_adapter - Fine-Tuned weights for Llama3.1 model

• 📁Results – Model evaluation metrics and outputs

• 📁Visuals – Diagrams, visual abstracts, and illustrations

  • 📁EDA – Exploratory data analysis visualizations
  • 📁Flowcharts – System architecture and end-to-end pipeline diagrams

• 📁Resources – Supplementary materials and external references

🎓 Team Members

• Yossef Okropiridze
• Ohad Bitton
• Michael Naftalishen