LRAGE: Legal Retrieval Augmented Generation Evaluation Tool

LRAGE (Legal Retrieval Augmented Generation Evaluation, pronounced as 'large') is an open-source toolkit designed to evaluate Large Language Models (LLMs) in a Retrieval-Augmented Generation (RAG) setting, specifically tailored for the legal domain.

This repository contains pointers to datasets and code used in LRAGE: Legal Retrieval Augmented Generation Evaluation.

Code: https://github.com/hoorangyee/LRAGE

Features

• Legal Domain Focused Evaluation: LRAGE is specifically developed for evaluating LLMs in a RAG setting with datasets and document collections from the legal domain, such as Pile-of-law, LegalBench, LawBench, KBL, and Legal RAG Benchmarks.

• Pre-compiled indexes for the legal domain: Comes with pre-generated BM25 indices and embeddings for Pile-of-law, reducing the setup effort for researchers.

• Retriever & Reranker Integration: Easily integrate and evaluate different retrievers and rerankers. LRAGE modularizes retrieval and reranking components, allowing for flexible experimentation.

• smolagents Integration: Seamlessly integrates with the smolagents framework, enabling evaluation of autonomous agents in legal RAG scenarios. This allows researchers to assess how agent-based approaches perform in complex legal tasks requiring multi-step reasoning.

• LLM-as-a-Judge: A feature where LLMs are used to evaluate the quality of LLM responses on an instance-by-instance basis, using customizable rubrics within the RAG setting.

• Graphical User Interface: A GUI demo for intuitive usage, making the tool accessible even for those who are not deeply familiar with command-line interfaces.