Carbon-Footprint-Aware System for Optimal LLM Selection and Prompt Compression

This project is an LLM optimization and evaluation system designed to reduce the carbon footprint and computational cost of Large Language Model (LLM) inference. It accomplishes this through intelligent prompt compression and dynamic model routing based on prompt complexity, while tracking the energy consumption and carbon emissions of the generation process.

Features

• Prompt Compression: Utilizes LLMLingua to effectively compress input prompts, reducing the number of tokens without losing the semantic meaning or intent.
• Dynamic Model Routing: Analyzes the complexity of the input prompt using Nvidia NeMo Curator and intelligently routes the prompt to the most suitable LLM (e.g., Gemini 1.5 Flash 8B for low complexity, Gemini 1.5 Pro for high complexity) to optimize resource usage.
• Eco-Tracking: Integrates with EcoLogits to track and estimate the energy consumption (kWh) and carbon footprint (kg CO2eq) for each request, allowing for direct comparison between the original and compressed prompt requests.
• Automated Judging: Uses an independent Groq model (llama-3.3-70b-versatile) to evaluate and compare the quality of the responses generated from the original and compressed prompts, ensuring that quality is maintained while saving resources.
• RESTful API Backend: A FastAPI powered backend exposing the /evaluate endpoint for seamless integration.
• Interactive UI: A React/Vite-based frontend for interacting with the evaluation system.

Project Structure

• server.py: The main FastAPI server exposing the evaluation endpoint.
• compress_prompt.py: Core logic for prompt compression using LLMLingua-2 and EcoLogits.
• llm_router.py: Handles analyzing prompt complexity and routing to the optimal LLM.
• Nvidia prompt class/: Contains scripts and logic for utilizing NeMo Curator for prompt complexity analysis.
• frontend/: The React (Vite) frontend application.

Prerequisites

• Python 3.9+
• Node.js (for the frontend)
• API Keys:
  • Gemini API Key (Google GenAI)
  • Groq API Key

Setup & Installation

1. Backend Setup

1. Clone the repository and navigate to the project directory:

   cd Carbon-Footprint-Aware-System-for-Optimal-LLM-Selection-and-Prompt-Compression
   

2. Create a virtual environment and activate it:

   python -m venv venv
   source venv/bin/activate  # On Windows use: venv\Scripts\activate
   

3. Install the required Python packages:

   pip install -r requirements.txt
   

4. Configure Environment Variables: Copy the example environment file and fill in your API keys.

   cp .env.example .env
   

   Open .env and add:

   GEMINI_API_KEY=your_gemini_api_key_here
   GROQ_API_KEY=your_groq_api_key_here
   

2. Frontend Setup

1. Navigate to the frontend directory:

   cd frontend
   

2. Install the Node.js dependencies:

   npm install
   

3. Configure Frontend Environment Variables: Copy .env.example to .env inside the frontend folder.

   cp .env.example .env
   

   Set the API URL for the backend (usually http://localhost:8000 for local development):

   VITE_API_URL=http://localhost:8000
   

Usage

Running Locally

1. Start the Backend Server: From the root project directory:

   python server.py
   

   The backend will start at http://0.0.0.0:8000.

2. Start the Frontend Development Server: From the frontend/ directory:

   npm run dev
   

   The UI will be accessible typically at http://localhost:5173.

Endpoint

The backend provides a main endpoint:

• POST /evaluate: Accepts a JSON payload {"prompt": "Your text here"}. Returns a comprehensive evaluation comparing the original prompt's response against the optimized, compressed route, including token counts, savings, and an AI judge's rating.

Deployment

For detailed instructions on deploying the backend (Docker + Render) and frontend (Vercel), please refer to the DEPLOYMENT.md guide.

Logging

The system logs compression ratios, origin/compressed token counts, and carbon emission stats to compression_logs.txt locally for analysis.