Aligning Human Code Preferences via Adaptive Preference Optimization

This repository contains the implementation and experimental code for our paper.

Supported Models

• Qwen: Qwen2.5-Coder-1.5B-Instruct
• Qwen-7B: Qwen2.5-Coder-7B-Instruct
• DeepSeek: deepseek-coder-1.3b-instruct
• DeepSeek-6.7B: deepseek-coder-6.7b-instruct
• Llama: Llama-3.2-1B-Instruct
• Magicoder-6.7B: Magicoder-DS-6.7B

Evaluation Tasks

The framework evaluates models across five key dimensions of code quality:

• correctness: Code Correctness
• security: Code Security
• smell: Code Smell
• efficiency: Code Efficiency
• complexity: Code Complexity
• conciseness: Code Conciseness

Installation

Prerequisites

pip install torch transformers vllm datasets tqdm python-dotenv

Environment Setup

Create a .env file in the root directory with your model access configurations if needed.

Usage

1. Model Training

Train models using different optimization approaches:

Supervised Fine-Tuning (SFT)

python src/sft.py --model Qwen --task smell

Direct Preference Optimization (DPO)

python src/dpo.py --model Qwen --task smell

Combined SFT + DPO

python src/sft+dpo.py --model Qwen --task smell

Attention-based Preference Optimization (APO)

python src/apo.py --model Qwen --task smell

2. Code Generation

Generate code samples using trained models:

python generate_code.py --model Qwen --task smell --sample 5 --type SFT

Parameters:

• --model: Model to use (Qwen, DeepSeek, Llama, etc.)
• --task: Evaluation task (correctness, security, smell, efficiency, complexity, conciseness)
• --sample: Number of samples to generate per problem (default: 5)
• --type: Model type (Ori, SFT, DPO, SFT+DPO, APO)

3. Evaluation

Evaluate generated code solutions:

python eval/apps/eval/test_one_solution.py --save ./results/smell/Qwen-SFT --print_results

For tasks other than "correctness", run additional validation:

python valid.py --save ./results/smell/Qwen-SFT --task smell

Project Structure

APO-Replication/
├── src/                   # Training scripts
│   ├── sft.py             # Supervised Fine-Tuning
│   ├── dpo.py             # Direct Preference Optimization
│   ├── sft+dpo.py         # Combined SFT+DPO
│   └── apo.py             # Adaptive Preference Optimization
├── eval/                  # Evaluation scripts
│   └── apps/              # APPS benchmark evaluation
├── datasets/              # Training and evaluation datasets
├── results/               # Generated results and outputs
├── saved_models/          # Trained model checkpoints
├── generate_code.py       # Code generation script
├── model.py               # Model wrapper and configuration
├── valid.py               # Validation script
├── testing_util.py        # Testing utilities

Results

Results are saved in the ./results/{task}/{model}-{type}/ directory structure:

• all_codes.json: Generated code samples
• all_results.json: Evaluation results
• all_results_{task}.json: Task-specific evaluation results