Project:

Development:

Contents

• Overview
• Evaluations
• Installation
• Development
• Additional Information
• License

Overview

mellea-lrc checks whether the citations in a legal filing are valid and checks specificially for AI-hallucinated citations. It reads a document, finds every citation in it, and uses a CourtListener backend to check for citation existence and to match how the filing cites it — case name, court, year, and the proposition a pinpoint citation is offered for.

Three layers run in order, each consuming what the last produced:

Every citation keeps a span into the preprocessed text, and each validation step is recorded as its own node, so a verdict can be traced back to the characters that produced it. Extraction is deterministic and offline; validation only needs a CourtListener key and a model endpoint.

Usage of Mellea in mellea-lrc

Mellea, an open source library from IBM is used to improve the robustness of this three layer process through its use of an Instruct-Verify-Repair loop. Mellea is used to instruct a large language model with a specific prompt, validate the output against defined requirements, and then repair by automatically retrying the prompt with additional context. This loop enables more effective usage of Small Language Models (SLMs) like Granite, which can be more easily run from a consumer-grade laptop or desktop.

How to use mellea-lrc

Validation reaches CourtListener and a model, so set those up first:

cp .env.example .env    # then fill in the values below

Nothing reads .env implicitly, so the commands below are run through uv run --env-file .env. Exporting the variables into your shell works too.

To obtain a CourtListener API token, refer to Getting an API token.

To use the application in any serious manner, or to reproduce the evaluations, refer to Rate limits — a free-tier token will not carry a full run.

Installing the package provides a mellea-lrc command. It parses the citations out of a source and validates each one. examples/sample-filing.txt is a paragraph to try it on — one citation, stating one proposition the cited page actually supports:

uv run --env-file .env mellea-lrc validate \
  --from-file examples/sample-filing.txt --output validated.json

Parsed 1 citations; validating
=== INFO ====== Sampling was successful.
=== INFO ====== Sampling was successful.
Wrote validated.json

Without --from-file the argument is the text itself — the same paragraph, pasted straight in:

uv run --env-file .env mellea-lrc validate "The District defends its assignment plan on a single ground: that the two campuses it operates receive identical funding, identical staffing, and identical facilities, so no student is disadvantaged by being sent to one rather than the other. That defense is foreclosed. Segregating public school children by race denies them equal educational opportunity even when the physical facilities and other tangible factors at each school are equal. See Brown v. Board of Education, 347 U.S. 483, 495 (1954)." --output validated.json

--output takes the filename; a bare one after the text is rejected, since that slot is the text itself.

validated.json holds one node per check, in the order they ran. On this paragraph every one of them agrees:

The last one is the substantive check — whether the cited page carries the proposition the citing sentence attributes to it — and it has to quote the page to say so:

{
  "node_type": "MelleaPinpointCheckNode",
  "outcome": "supports",
  "evidence_quote": "Separate educational facilities are inherently unequal.",
  "evidence_span": { "start": 172, "end": 227 },
  "evidence_match_method": "exact",
  "evidence_match_score": 1.0
}

A verdict the model cannot ground in a quote from the page is discarded rather than reported, which is why evidence_span is always an offset into the text that was actually retrieved.

Two things worth knowing. Keep a citation on one line: a break between the reporter and the page — 347 U.S. / 483 — is damage the extractor does not repair, and the citation is missed rather than misread. And pass the argument the surrounding sentences, not the citation alone; the proposition being checked is read out of the text around the citation, so there has to be one there.

The Python API

Extraction takes a str as content or a Path as a location:

from pathlib import Path

from mellea_lrc.extraction import extract

document = extract(Path("filing.pdf"))
for citation in document.full_citations:
    print(citation.matched_text, citation.locator_span)

Validation is async, and reads COURTLISTENER_API_TOKEN and MELLEA_LRC_LLM_* from the environment:

import asyncio

from mellea_lrc.validation import validate_document

validated = asyncio.run(validate_document(document))
for entry in validated.citations:
    print(entry.citation_id, entry.aggregation)

Evaluations

Extraction and validation are evaluated separately against the frozen false-citation-bench dataset. Each evaluator reads a small JSONL run artifact, so any system can be scored, not only this one. See evaluations/.

Installation

Development

Please visit Contributing and Development for information on contributing to this project.

Additional Information

Additional information can be found at these locations.

License

mellea-lrc is licensed under the MIT license.