Knowledge Distillation & Explainability for Code Generation models

This repository contains two main components:

1. all-in-one
   A data-collection, knowledge distillation and explainability toolkit.
2. humaneval-benchmark
   A simple benchmarking script for code generation models on the HumanEval dataset.

---

Table of Contents

• Features
• Installation
• Usage
  • Running the all-in-one pipeline
  • Running the HumanEval benchmark
• Project Structure

---

Features

• aio.py – Main entry point for the all-in-one data-collection, knowledge distillation and explainability toolkit pipeline.
• collect.py & collect_infidelities.py – Helpers to scrape/collect raw data and measure infidelities.
• core/ – All core logic lives here:
  • distillation.py
  • explainability.py & workers
  • explainable_finetuning.py
  • Utility functions & prompts
• humaneval-benchmark/ – Baseline teacher vs. student evaluator on the HumanEval JSONL dataset.

---

Installation

1. Clone this repo:

   git clone https://github.com/SW-P6/P6-Final.git
   cd P6-Final
   

2. Install requirements for all-in-one:

   cd all-in-one
   pip install -r requirements.txt
   

3. Install requirements for humaneval-benchmark:

   cd humaneval-benchmark
   pip install -r requirements.txt
   

---

Usage

Running the all-in-one pipeline

1. Make sure you have your API keys/tokens set up as needed from HuggingFace.

2. Feel free to edit the contents of run.sh to your requirements.

3. Then from the repository root:

   cd all-in-one
   sh run.sh
   

   This will:

   • Launch aio.py (the main driver)
   • Invoke all core modules under core/ for distillation, explainability, etc.

Running the HumanEval benchmark

1. Feel free to edit the hard-coded contents of humaneval-benchmark.py to your requirements.
2. From the repository root:

   cd humaneval-benchmark
   python humaneval-benchmark.py
   

3. The script then reads HumanEval.jsonl and reports the code generation model's pass@k metrics.

---

Project Structure

.
├── README.md
├── all-in-one
│   ├── aio.py                    # ← Main entry point
│   ├── collect.py                # ← Data collection helper
│   ├── collect_infidelities.py   # ← Infidelity measurement helper
│   ├── core                      # ← Core logic modules
│   │   ├── distillation.py
│   │   ├── explainability.py
│   │   ├── explainability_workers.py
│   │   ├── explainable_finetuning.py
│   │   ├── functions.py
│   │   ├── prompts.py
│   │   ├── setup.py
│   │   └── vllm.py
│   ├── requirements.txt          # ← Python dependencies
│   └── run.sh                    # ← Shell script to launch the pipeline
└── humaneval-benchmark
    ├── HumanEval.jsonl           # ← HumanEval benchmark dataset
    ├── humaneval-benchmark.py    # ← Benchmarking script
    └── requirements.txt          # ← Python dependencies