Mil

Official implementation of Mil: Cost-guided Minimum Makespan Scheduling for Applications of Multiple LLMs. Mil is built on top of vLLM.

Paper  ·  DOI  ·  Poster PDF  ·  Editable poster

Poster

Click the poster to open the full-resolution PDF.
View PDF  ·  Download editable PowerPoint

Installation

Install the version of vLLM included in this repository before running Mil.

Repository structure

• benchmarks/ contains the core implementation of Mil.
• benchmarks/MY_README.md describes the code structure.
• benchmarks/add_a_new_model.md explains how to build the cost model for a new model.
• technical_report/technical_report.pdf contains the technical report.

Reproducing the paper's experiments

Use benchmarks/run_end2end_exp.sh to run the end-to-end experiments reported in the paper.