README
ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India (AAAI 2025)
[π GitHub] β’ [π ArXiv] β’ [π€ HF Results] β’ [π€ HF Dataset]
This is the official implementation of the paper:
Shubham Kumar Nigam, Tanuj Tyagi, Siddharth Shukla, Aditya Kumar Guru, Balaramamahanthi Deepak Patnaik, Danush Khanna, Noel Shallum, Kripabandhu Ghosh, and Arnab Bhattacharya:
ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India (AAAI 2025)
Overview
ReGal presents an early exploration of Reinforcement Learning (RL) methodologies for legal AI in the Indian context. We introduce a framework that integrates Multi-Task Instruction Tuning with Reinforcement Learning from AI Feedback (RLAIF) using Proximal Policy Optimization (PPO). Our approach is evaluated across two critical legal tasks: (i) Court Judgment Prediction and Explanation (CJPE), and (ii) Legal Document Summarization.
While ReGal underperforms compared to supervised and proprietary models, it provides valuable insights into the challenges of applying RL to legal texts, including reward model alignment, legal language complexity, and domain-specific adaptation. This work establishes a foundation for future improvements in RL-based legal AI systems.
If you have any questions about this work, please open a GitHub issue or email the authors at:
shubhamkumarnigam@gmail.com, tanujtyagiofficial@gmail.com, danush.s.khanna@gmail.com
Key Contributions
- First PPO Application: One of the first applications of PPO-based reinforcement learning in Indian legal judgment prediction and summarization.
- Comprehensive Analysis: Empirical and qualitative evidence on the limitations of PPO for legal NLP tasks.
- Future Directions: A clear path for more effective legal-AI pipelines integrating RLHF, human feedback, and domain-adapted modeling.
Getting Started
General Instructions
Ensure you have the necessary hardware and software requirements in place to replicate our experimental setup. Follow the steps below to configure your environment for optimal performance.
Recommended Hardware Configuration
Hardware Specifications:
- Two cores of NVIDIA A100-PCIE-40GB with 126GB RAM for instruction fine-tuning and PPO training.
- Alternatively, Vast.ai A100 80GB GPU rental (~$100 for complete training).
- Google Colab Pro with A100 accelerator is sufficient for inference and baseline experiments.
Recommended Software Configuration
Software Setup:
- Python 3.8+
- PyTorch 2.0+
- Transformers 4.30+
- Additional dependencies for PPO training (TRL, bitsandbytes)
- Install necessary drivers and libraries for GPU acceleration.
Installation
git clone https://github.com/ShubhamKumarNigam/ReGal.git
cd ReGal
pip install -r requirements.txt
Tasks & Datasets
Task 1: Court Judgment Prediction and Explanation (CJPE)
Task 1A - Judgment Prediction: Predict whether an appeal/petition from the Supreme Court of India was accepted (1) or rejected (0).
Task 1B - Rationale Explanation: Generate natural language explanations supporting the predicted outcome.
Dataset - PredEx: The largest annotated dataset for Indian legal judgment prediction and explanation with 15,222 Supreme Court judgment documents.
| Metric | Value |
|---|---|
| Train Documents | 12,178 |
| Test Documents | 3,044 |
| Avg. Tokens per Document | 4,586 |
| Max Tokens | 117,733 |
| Acceptance Rate | 53.44% |
Task 2: Legal Judgment Summarization
Generate concise, abstractive summaries capturing essential components (background, legal issues, arguments, verdict) from full judgment texts.
Dataset - In-Abs: Expert-curated abstractive summaries from Indian Supreme Court judgments.
| Metric | Value |
|---|---|
| Total Documents | 7,130 |
| Train/Test Split | 7,030 / 100 |
| Avg. Document Size | 4,376.98 words |
| Avg. Summary Size | 842.52 words |
| Compression Ratio | 0.235 |
Methodology
Two-Stage Approach
-
Stage 1: Supervised Fine-Tuning (SFT)
- Fine-tune Llama-2-7B on labeled legal data
- Serves as reference policy (Ο_SFT) for PPO training
-
Stage 2: Proximal Policy Optimization (PPO)
- Use task-specific reward models to guide policy optimization
- Minimize PPO loss to align outputs with desired legal outcomes
- Apply KL-divergence penalty to prevent deviation from SFT baseline
Reward Models
- CJPE Reward Model: Fine-tuned InLegalBERT classifier (binary rewards: 1 for correct prediction, 0 otherwise)
- Summarization Reward Model: ROUGE-based n-gram overlap + semantic similarity scoring
Training Configuration
| Hyperparameter | Value |
|---|---|
| Base Model | Llama-2-7B |
| Learning Rate | 1.41e-5 |
| Batch Size | 4 |
| Mini-batch Size | 2 |
| PPO Epochs | 1 |
| Output Length | 100-500 tokens |
| Clipping Parameter (Ξ΅) | 0.1 |
| GPU | NVIDIA A100 80GB |
| Total Training Cost | ~$100 |
Evaluation Metrics
Lexical-Based Metrics
- ROUGE-1/2/L: Recall-based n-gram overlap
- BLEU: Precision-based evaluation
- METEOR: Synonym and stemming-aware comparison
Semantic-Based Metrics
- BERTScore: Semantic similarity using contextual embeddings
- BLANC: Semantic relevance assessment
Results
Judgment Prediction and Explanation Performance
Legal Summarization Performance
In-Abs Summarization Dataset
Key Findings & Challenges
Reasons for Underperformance
- Objective Mismatch: SFT baseline not fully optimized for legal reasoning
- Reward Model Limitations: Difficulty capturing nuanced legal interpretations
- Legal Complexity: Long, intricate documents with rich semantic references
- Training Data Constraints: Limited diversity in legal reasoning patterns
- Hallucination Issues: Model generates plausible but factually incorrect outputs
- Domain Pretraining Gap: Lack of deep domain adaptation compared to GPT-3.5
- Hyperparameter Sensitivity: Suboptimal tuning of learning rates and penalties
- Model Architecture: Llama-2-7B may be undersized for complex legal tasks
Hallucination Analysis
The ReGal framework exhibits significant hallucination issues, particularly when:
- Input facts are sparse or ambiguously phrased
- The model over-optimizes for weak reward patterns
- Outputs mimic style without substantive accuracy
Examples include fabricated legal principles, invented precedent citations, and claims unsupported by source documents.
Ablation Study
Base Model Variants
- Phi-3 Mini: Too small for complex legal texts; severe performance degradation
- Llama-2-7B (Pretrained): Insufficient without legal domain fine-tuning
- Llama-2-7B SFT: Optimal balance of capacity and domain adaptation
Reward Model Variants
- Task-Specific Reward Model: Best performance with domain-aligned scoring
- General InLegalBERT: Noisier feedback; degraded PPO optimization
Conclusion: Success hinges on strong initialization and precisely aligned reward functions.
Future Work
- Better Reward Models: Develop hallucination-aware and factuality-constrained reward signals
- Domain Pretraining: Leverage domain-adaptive pretraining strategies for legal corpora
- Human-in-the-Loop: Integrate human feedback to improve learning signals
- Advanced Architectures: Explore larger models and specialized legal LLMs
- Factuality Constraints: Augment PPO with stronger constraints on factual accuracy
- Cross-Jurisdiction Transfer: Extend to multiple legal systems and languages
Citation
If you use this work, please cite our paper:
@article{nigam2025regal,
title={ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India},
author={Nigam, Shubham Kumar and Tyagi, Tanuj and Shukla, Siddharth and Guru, Aditya Kumar and Patnaik, Balaramamahanthi Deepak and Khanna, Danush and Shallum, Noel and Ghosh, Kripabandhu and Bhattacharya, Arnab},
journal={arXiv preprint arXiv:2512.18014},
year={2025}
}
License
This project is licensed under the MIT License - see the LICENSE file for details.
Acknowledgments
- IIT Kanpur, Manipal University Jaipur, Symbiosis Law School Pune, and IISER Kolkata
- Funded through academic research initiatives
- Special thanks to the legal experts who annotated the datasets
Contact
For questions or collaborations, reach out to:
- Shubham Kumar Nigam: shubhamkumarnigam@gmail.com
- Danush Khanna: danush.s.khanna@gmail.com
- Kripabandhu Ghosh: kripaghosh@iiserkol.ac.in
- Arnab Bhattacharya: arnabb@cse.iitk.ac.in
Repository: GitHub - ReGal
Contributors
Similar repos
Languages
Python
100.0%
README
ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India (AAAI 2025)
[π GitHub] β’ [π ArXiv] β’ [π€ HF Results] β’ [π€ HF Dataset]
This is the official implementation of the paper:
Shubham Kumar Nigam, Tanuj Tyagi, Siddharth Shukla, Aditya Kumar Guru, Balaramamahanthi Deepak Patnaik, Danush Khanna, Noel Shallum, Kripabandhu Ghosh, and Arnab Bhattacharya:
ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India (AAAI 2025)
Overview
ReGal presents an early exploration of Reinforcement Learning (RL) methodologies for legal AI in the Indian context. We introduce a framework that integrates Multi-Task Instruction Tuning with Reinforcement Learning from AI Feedback (RLAIF) using Proximal Policy Optimization (PPO). Our approach is evaluated across two critical legal tasks: (i) Court Judgment Prediction and Explanation (CJPE), and (ii) Legal Document Summarization.
While ReGal underperforms compared to supervised and proprietary models, it provides valuable insights into the challenges of applying RL to legal texts, including reward model alignment, legal language complexity, and domain-specific adaptation. This work establishes a foundation for future improvements in RL-based legal AI systems.
If you have any questions about this work, please open a GitHub issue or email the authors at:
shubhamkumarnigam@gmail.com, tanujtyagiofficial@gmail.com, danush.s.khanna@gmail.com
Key Contributions
- First PPO Application: One of the first applications of PPO-based reinforcement learning in Indian legal judgment prediction and summarization.
- Comprehensive Analysis: Empirical and qualitative evidence on the limitations of PPO for legal NLP tasks.
- Future Directions: A clear path for more effective legal-AI pipelines integrating RLHF, human feedback, and domain-adapted modeling.
Getting Started
General Instructions
Ensure you have the necessary hardware and software requirements in place to replicate our experimental setup. Follow the steps below to configure your environment for optimal performance.
Recommended Hardware Configuration
Hardware Specifications:
- Two cores of NVIDIA A100-PCIE-40GB with 126GB RAM for instruction fine-tuning and PPO training.
- Alternatively, Vast.ai A100 80GB GPU rental (~$100 for complete training).
- Google Colab Pro with A100 accelerator is sufficient for inference and baseline experiments.
Recommended Software Configuration
Software Setup:
- Python 3.8+
- PyTorch 2.0+
- Transformers 4.30+
- Additional dependencies for PPO training (TRL, bitsandbytes)
- Install necessary drivers and libraries for GPU acceleration.
Installation
git clone https://github.com/ShubhamKumarNigam/ReGal.git
cd ReGal
pip install -r requirements.txt
Tasks & Datasets
Task 1: Court Judgment Prediction and Explanation (CJPE)
Task 1A - Judgment Prediction: Predict whether an appeal/petition from the Supreme Court of India was accepted (1) or rejected (0).
Task 1B - Rationale Explanation: Generate natural language explanations supporting the predicted outcome.
Dataset - PredEx: The largest annotated dataset for Indian legal judgment prediction and explanation with 15,222 Supreme Court judgment documents.
| Metric | Value |
|---|---|
| Train Documents | 12,178 |
| Test Documents | 3,044 |
| Avg. Tokens per Document | 4,586 |
| Max Tokens | 117,733 |
| Acceptance Rate | 53.44% |
Task 2: Legal Judgment Summarization
Generate concise, abstractive summaries capturing essential components (background, legal issues, arguments, verdict) from full judgment texts.
Dataset - In-Abs: Expert-curated abstractive summaries from Indian Supreme Court judgments.
| Metric | Value |
|---|---|
| Total Documents | 7,130 |
| Train/Test Split | 7,030 / 100 |
| Avg. Document Size | 4,376.98 words |
| Avg. Summary Size | 842.52 words |
| Compression Ratio | 0.235 |
Methodology
Two-Stage Approach
-
Stage 1: Supervised Fine-Tuning (SFT)
- Fine-tune Llama-2-7B on labeled legal data
- Serves as reference policy (Ο_SFT) for PPO training
-
Stage 2: Proximal Policy Optimization (PPO)
- Use task-specific reward models to guide policy optimization
- Minimize PPO loss to align outputs with desired legal outcomes
- Apply KL-divergence penalty to prevent deviation from SFT baseline
Reward Models
- CJPE Reward Model: Fine-tuned InLegalBERT classifier (binary rewards: 1 for correct prediction, 0 otherwise)
- Summarization Reward Model: ROUGE-based n-gram overlap + semantic similarity scoring
Training Configuration
| Hyperparameter | Value |
|---|---|
| Base Model | Llama-2-7B |
| Learning Rate | 1.41e-5 |
| Batch Size | 4 |
| Mini-batch Size | 2 |
| PPO Epochs | 1 |
| Output Length | 100-500 tokens |
| Clipping Parameter (Ξ΅) | 0.1 |
| GPU | NVIDIA A100 80GB |
| Total Training Cost | ~$100 |
Evaluation Metrics
Lexical-Based Metrics
- ROUGE-1/2/L: Recall-based n-gram overlap
- BLEU: Precision-based evaluation
- METEOR: Synonym and stemming-aware comparison
Semantic-Based Metrics
- BERTScore: Semantic similarity using contextual embeddings
- BLANC: Semantic relevance assessment
Results
Judgment Prediction and Explanation Performance
Legal Summarization Performance
In-Abs Summarization Dataset
Key Findings & Challenges
Reasons for Underperformance
- Objective Mismatch: SFT baseline not fully optimized for legal reasoning
- Reward Model Limitations: Difficulty capturing nuanced legal interpretations
- Legal Complexity: Long, intricate documents with rich semantic references
- Training Data Constraints: Limited diversity in legal reasoning patterns
- Hallucination Issues: Model generates plausible but factually incorrect outputs
- Domain Pretraining Gap: Lack of deep domain adaptation compared to GPT-3.5
- Hyperparameter Sensitivity: Suboptimal tuning of learning rates and penalties
- Model Architecture: Llama-2-7B may be undersized for complex legal tasks
Hallucination Analysis
The ReGal framework exhibits significant hallucination issues, particularly when:
- Input facts are sparse or ambiguously phrased
- The model over-optimizes for weak reward patterns
- Outputs mimic style without substantive accuracy
Examples include fabricated legal principles, invented precedent citations, and claims unsupported by source documents.
Ablation Study
Base Model Variants
- Phi-3 Mini: Too small for complex legal texts; severe performance degradation
- Llama-2-7B (Pretrained): Insufficient without legal domain fine-tuning
- Llama-2-7B SFT: Optimal balance of capacity and domain adaptation
Reward Model Variants
- Task-Specific Reward Model: Best performance with domain-aligned scoring
- General InLegalBERT: Noisier feedback; degraded PPO optimization
Conclusion: Success hinges on strong initialization and precisely aligned reward functions.
Future Work
- Better Reward Models: Develop hallucination-aware and factuality-constrained reward signals
- Domain Pretraining: Leverage domain-adaptive pretraining strategies for legal corpora
- Human-in-the-Loop: Integrate human feedback to improve learning signals
- Advanced Architectures: Explore larger models and specialized legal LLMs
- Factuality Constraints: Augment PPO with stronger constraints on factual accuracy
- Cross-Jurisdiction Transfer: Extend to multiple legal systems and languages
Citation
If you use this work, please cite our paper:
@article{nigam2025regal,
title={ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India},
author={Nigam, Shubham Kumar and Tyagi, Tanuj and Shukla, Siddharth and Guru, Aditya Kumar and Patnaik, Balaramamahanthi Deepak and Khanna, Danush and Shallum, Noel and Ghosh, Kripabandhu and Bhattacharya, Arnab},
journal={arXiv preprint arXiv:2512.18014},
year={2025}
}
License
This project is licensed under the MIT License - see the LICENSE file for details.
Acknowledgments
- IIT Kanpur, Manipal University Jaipur, Symbiosis Law School Pune, and IISER Kolkata
- Funded through academic research initiatives
- Special thanks to the legal experts who annotated the datasets
Contact
For questions or collaborations, reach out to:
- Shubham Kumar Nigam: shubhamkumarnigam@gmail.com
- Danush Khanna: danush.s.khanna@gmail.com
- Kripabandhu Ghosh: kripaghosh@iiserkol.ac.in
- Arnab Bhattacharya: arnabb@cse.iitk.ac.in
Repository: GitHub - ReGal
Similar repos
Contributors
Languages
Python
100.0%