AgentModernize

Preserving Business Logic in Legacy Modernization with Multi-Agent LLMs and Behavioral Specification Graphs

AgentModernize is a multi-agent LLM framework that treats legacy modernization as a behavioral preservation problem. Four specialized agents handle extraction, specification, code generation, and equivalence validation, connected through Behavioral Specification Graphs (BSGs) that make extracted business logic explicit and inspectable before any code is generated.

For methodology, BER tables, and discussion, cite https://arxiv.org/abs/2605.17535. This repository implements the evaluation protocol described in that paper.

Architecture

Legacy Artifact Bundle
        |
  Agent 1: Legacy Analyzer --> Business Rule Inventory
        |
  Agent 2: Spec Generator  --> Behavioral Specification Graph (BSG)  [trust boundary]
        |
  Agent 3: Transformer     --> Modernized Service
        |                          ^
  Agent 4: Validator        -------| (retry Agent 3: Transformer with feedback, ≤3×)
        |
  Equivalence Report

Benchmark: LegacyModernize-8

Eight legacy modernization scenarios (COBOL / PL-SQL) spanning telecom and banking domains, each with:

• Legacy source code (100–310 LOC)
• Gold-standard behavioral specs (curated; AI-assisted drafting, human-reviewed)
• Withheld behavioral test suites for fair evaluation

Project Structure

src/              # Pipeline implementation (4 agents + orchestrator)
benchmark/        # LegacyModernize-8 scenarios (S1–S8, COBOL/PL-SQL + gold-standard tests)
eval_bsg.py       # BSG extraction quality evaluation
run_experiment.py # Single-scenario experiment runner
run_fair_eval_existing.py    # Fair evaluation across scenarios (existing `results/` layouts)
run_full_baseline_comparison.py  # SP-LLM / CoT-LLM / AM comparison
run_model_comparison.py      # GPT-4o-mini vs GPT-4o vs GPT-5.3-codex (see code for exact API model id)
run_codex_comparison.py      # Frontier model study

Quick Start

# Install dependencies
pip install -r requirements.txt

# Set your OpenAI API key
export OPENAI_API_KEY=your-key-here

# Run a single scenario
python run_experiment.py --scenario S1

# Re-run fair evaluation on existing result folders (paper-style tables: 8 scenarios,
# 3 trials, model layouts mini / gpt4o / codex)
python run_fair_eval_existing.py --model all --trials 3

# Or: run AM + baselines + fair eval in one driver (single pass per cell; default mini layout)
python run_experiment.py --fair-eval-all

# Run 3-model comparison
python run_model_comparison.py

On Windows PowerShell (same Python commands as above after setting the key):

pip install -r requirements.txt
$env:OPENAI_API_KEY = "your-key-here"
python run_experiment.py --scenario S1
python run_fair_eval_existing.py --model all --trials 3
python run_experiment.py --fair-eval-all
python run_model_comparison.py

Reproducibility

Citation

If you use AgentModernize or LegacyModernize-8 in your research, please cite:

@misc{ahmed2026agentmodernize,
  title={AgentModernize: Preserving Business Logic in Legacy Modernization with Multi-Agent LLMs and Behavioral Specification Graphs},
  author={Ahmed, Sheikh Nazib and Galib, Marnim},
  year={2026},
  eprint={2605.17535},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  url={https://arxiv.org/abs/2605.17535}
}

License

MIT