[ASPLOS'26] HILOS: A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs

This artifact provides the implementation of HILOS, a high-throughput near-storage processing system designed for batched LLM inference. We provide the complete source code and a pre-configured server to facilitate the setup and reproduction of our experimental results.

📰 News

• 🐝 [26/11/26] HILOS has been accepted at ASPLOS'26.

Table of Contents

• Abstract
• Overview
• Getting Started
  • Hardware Dependencies
  • Software Dependencies
• Installation
• Experiment Workflow
• Notes
• License
• Citation
• Contact

Abstract

This artifact supports:

1. FPGA Binary Synthesis: Generation of FPGA binaries using High-Level Synthesis (HLS)
2. Inference Pipeline Deployment: Full deployment of the batched LLM inference system

Overview

This repository contains two main components:

1. Step 1 (step1_hls_synth): FPGA binary synthesis using High-Level Synthesis (HLS)
2. Step 2 (step2_deployment): Batched LLM inference deployment using the generated FPGA binary

Important:

• If you want to directly start without Step 1, you can use the provided binary files in step2_deployment/bins/
• To reproduce the full process, complete Step 1 before proceeding to Step 2
• Each folder contains its own detailed README with specific instructions

Getting Started

Hardware Dependencies

If you wish to reproduce the experiments on your own hardware, the following components are required:

• GPU: At least one NVIDIA GPU with support for GPUDirect Storage (GDS)
• Storage: At least one Samsung SmartSSD
• CPU: An Intel CPU is mandatory for SmartSSD compatibility

Software Dependencies

The environment relies on the following components:

1. Drivers and CUDA

• NVIDIA driver: Version 530 or higher
• CUDA/CuDNN toolkit: Version 12.1 recommended
• Ensure CUDA paths are correctly added to the environment variables

2. GPUDirect Storage (GDS)

• Install the NVIDIA GDS package appropriate for your system kernel

3. FPGA Tools

Install the following tools:

• Xilinx Vivado and Vitis HLS: Version 2023.1
• Xilinx Runtime (XRT): The XRT version matching the target SmartSSD platform
  • Download from the AMD Alveo download page
  • Important: A cold reboot is mandatory after installation to flash the SmartSSD

Source the settings scripts prior to execution:

source <path_to_Vitis_HLS>/2023.1/settings64.sh
source /opt/xilinx/xrt/setup.sh

Installation

We recommend using an Anaconda environment rather than Docker, as the peer-to-peer (P2P) feature of the SmartSSD is not supported in containers.

Step 1: Environment Setup

Ensure the Anaconda environment manager is installed. Create and configure the environment:

conda create -n asplos26_ae python=3.9 -y
conda activate asplos26_ae
conda install -c conda-forge uv
uv pip install flash-attn --no-build-isolation

Step 2: HLS Synthesis Dependencies

Navigate to the step1_hls_synth directory and install dependencies:

cd step1_hls_synth
bash install.sh
uv pip install -e .

Step 3: FPGA Bitstream Generation

Generate the FPGA bitstream:

cd tests/csrc
make xclbin
cp <path_to_xclbin>/*.xclbin ../../bins/

Note: This step may take several hours. If you want to skip this step, pre-generated binaries are available in the bins/ directory.

Step 4: Deployment Dependencies

Navigate to the step2_deployment directory and install requirements:

cd ../../..  # Return to HILOS root
cd step2_deployment
uv pip install -r requirements.txt

Experiment Workflow

The evaluation follows a two-step workflow:

Step 1: HLS Synthesis and Verification

See step1_hls_synth/README.md for detailed instructions on:

• Verifying HLS functionality
• Generating and testing FPGA binaries

Step 2: LLM Inference Deployment

See step2_deployment/README.md for detailed instructions on:

• Running the baseline (FlexGen)
• Running HILOS with FPGA acceleration
• Running HILOS with X-Cache optimization
• Comparing performance results

Quick Start: Use the provided run_all.sh script to automate the full evaluation pipeline:

cd step2_deployment
sh run_all.sh

For detailed arguments and procedures, please refer to the README files in each respective directory

Notes

• The current implementation has been tested primarily with the OPT model family
• Ensure all environment variables and paths are correctly set before running experiments
• For any issues during setup or execution, please check the detailed README files in each step's directory

License

This project is licensed under the MIT License - see the LICENSE file for details.

Citation

If you use this code in your research, please cite the following paper:

@inproceedings{hilos,
  title={HILOS: A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs},
  author={Jang, Hongsun and Song, Jaeyong and Shin, Changmin and Noh, Si Ung and Jung, Jaewon and Park, Jisung and Lee, Jinho},
  booktitle={Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)},
  year={2026}
}

Contact

For any questions or comments about the paper or the code, please contact [email protected]

Enjoy using HILOS, and feel free to contribute to the project by opening issues or submitting pull requests!