Vector Special Functional Unit (vSFU)

This repository explores algorithms and hardware for a vectorized Special Functional Unit (SFU) that approximates non-linear activation functions (SiLU, GELU, tanh, sigmoid, exp, …) with piecewise-linear (PWL) interpolation.

The repo is split into two parts:

• algo_eval/ — Python / PyTorch code that trains and evaluates PWL approximations of activation functions, and integrates them into networks (timm/ImageNet, HuggingFace/GLUE). This is the main focus of this README.
• hw/ — Verilog/CocoTB hardware implementation of the SFU. See the SFU Hardware Eval section near the bottom for setup notes.

---

References

This repository accompanies the following papers. Please cite them if you use this code:

• R. Andri, E. Reggiani, and L. Cavigelli, "Flex-SFU: Activation Function Acceleration With Nonuniform Piecewise Approximation," IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 44, no. 11, pp. 4236–4248, 2025. IEEE Xplore · DOI.
• E. Reggiani, R. Andri, and L. Cavigelli, "Flex-SFU: Accelerating DNN Activation Functions by Non-Uniform Piecewise Approximation," in Proc. 60th ACM/IEEE Design Automation Conference (DAC), 2023, pp. 1–6. IEEE Xplore · arXiv:2305.04546 · DOI.

BibTeX:

@article{andri2025flexsfu,
  author  = {Andri, Renzo and Reggiani, Enrico and Cavigelli, Lukas},
  title   = {{Flex-SFU}: Activation Function Acceleration with Nonuniform Piecewise Approximation},
  journal = {IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems},
  year    = {2025},
  volume  = {44},
  number  = {11},
  pages   = {4236--4248},
  doi     = {10.1109/TCAD.2025.3558140},
}

@inproceedings{reggiani2023flexsfu,
  author    = {Reggiani, Enrico and Andri, Renzo and Cavigelli, Lukas},
  title     = {{Flex-SFU}: Accelerating {DNN} Activation Functions by Non-Uniform Piecewise Approximation},
  booktitle = {Proceedings of the 60th ACM/IEEE Design Automation Conference (DAC)},
  year      = {2023},
  pages     = {1--6},
  doi       = {10.1109/DAC56929.2023.10247855},
}

---

Quickstart: train a PWL approximation of an activation

The entry point is algo_eval/pwl_eval2.py. Each activation has a config file in algo_eval/configs/; pass the config's basename (no .py) via --config.

# from the repository root
conda activate timm-sfu
python -m algo_eval.pwl_eval2 --config silu

This will, for every breakpoint count in num_points_range (default [4, 8, 16, 32, 64] from algo_eval/config_global.py):

1. fit a uniform PWL (equispaced breakpoints, learn only the y-values),
2. fit a flex PWL (learn breakpoint positions and y-values),
3. run an insertion/removal refinement loop on the flex model (iteratively delete the worst breakpoint and re-insert it where the error is largest), and
4. write CSV, TensorBoard logs, PDF plots, and a dill-pickled snapshot of every model.

Outputs land under algo_eval/out/:

Inspect with tensorboard --logdir algo_eval/out/tensorboard/.

---

CLI reference (pwl_eval2.py)

Examples:

# Only flex PWL, 8 and 16 breakpoints, MAE loss, custom lr
python -m algo_eval.pwl_eval2 --config gelu --prefix flex --numbp 8,16 --loss mae --lr 5e-2

# Uniform PWL only (skips insertion automatically)
python -m algo_eval.pwl_eval2 --config tanh --prefix unif

# Second-order pieces
python -m algo_eval.pwl_eval2 --config silu --secondorder

---

Supported activations (algo_eval/configs/)

Each config sets func, func_name, an interpolation interval [init_low, init_high], the input distribution (batch_gen_normal, batch_gen_linear), the boundary_function used outside that interval, and optionally lr_config.

To add a new activation, copy configs/silu.py and adapt func, func_name, init_low/init_high, the boundary function, and the input distribution.

---

The PWL modules

Three implementations live side by side:

1. pwl.PWL — the main differentiable PWL module.
   • flex_bp=False → uniform breakpoints (only y-values and the left/right ends are learned).
   • flex_bp=True → flex breakpoints (positions are learnable bp_param and y-values are learned jointly via gradient descent).
   • secondOrder=True → quadratic pieces; an extra valCenter parameter per piece is optimised with scipy.optimize.minimize_scalar.
   • Key methods: optimize(batch_gen, writer, epochs, optimizer, losstype, lr, …), measure_quality(batch_gen), plot(batch_gen, save=…, mark=…), insert_bp(idx, batch_gen, loss), remove_bp(idx, batch_gen), calculate_insertion_loss(batch_gen, loss), calculate_removal_loss(batch_gen, loss), save_current_state() / reload_from_saved_state().
2. pwl_muggeo.PWL_Muggeo — wraps the piecewise_regression package (Muggeo 2003) to estimate non-uniform breakpoints by iterative re-linearisation. Often produces better fits than gradient descent but can fail to converge for high breakpoint counts (pwl_eval2.py only runs it for npts ≤ 16).
3. pwl_utils.convert_Muggeo2PWL — converts a PWL_Muggeo instance into a PWL(flex_bp=True) so it can be further refined via SGD.

The exploratory notebook is algo_eval/EvalutionNotebook.ipynb; the playground for Muggeo is algo_eval/pwl_muggeo_playground.py.

---

Repository layout

algo_eval/
├── pwl_eval2.py            # MAIN training/eval script (use this)
├── pwl.py                  # PWL module (uniform / flex / 2nd-order)
├── pwl_muggeo.py           # Muggeo-based PWL module
├── pwl_utils.py            # Muggeo→PWL conversion
├── config_global.py        # Global defaults (num_points_range, …)
├── configs/                # One file per activation/config
├── eval_from_pickle.py     # Reload trained models, re-evaluate, re-save
├── net_eval.py             # Helpers to inspect activation types in a model
├── input_histogram.py      # Collect activation-input histograms from a model
├── grad_wrapper.py         # Autograd wrapper used by grad_* configs
├── grad_sfu/               # GELU_SFU hardware-kernel reference
├── hw_aware_exec.py        # Stub for HW-aware execution
├── simple_exp_sfu.py       # Standalone exp(x) SFU demo
├── tformers.py             # PWL_Softmax + transformer weight substitution
├── pwl_eval_imagenet.py    # ImageNet eval harness using pickled PWLs
├── scripts/
│   ├── timm_validate.py    # ImageNet validation wrapper for timm
│   ├── run_glue.py         # HF GLUE fine-tune with PWL activations
│   ├── glue.py             # GLUE metric helper
│   ├── collect_glue_accuracy.py
│   ├── create_2ndorder_pickles.py  # Upgrade 1st→2nd-order pickled models
│   ├── update_pickles_with_statedict.py
│   ├── pickle2dictth.py    # dill pickle → torch state_dict
│   └── minima_avoidance.py # TensorBoard-log analysis of local minima
├── timm_train/             # timm reference trainer (train.py + train_model.sh)
├── src/                    # Pre-trained pickles (1st/2nd order, flex/uniform)
├── out/                    # All run outputs (csv, pickles, plots, TB logs)
├── old/                    # Superseded scripts (quick-eval.py)
├── pwl_eval.py             # Older training loop (use pwl_eval2.py instead)
├── old-timm_validate.py    # Older timm validation interface
├── EvalutionNotebook.ipynb # Notebook for interactive analysis
└── transformers.sh         # Clone the patched HF transformers fork
slurm/
└── run_all.sh              # Sweep every config in configs/ via SLURM
hw/                         # Verilog SFU + CocoTB testbenches

---

Setup

Two environments are used, depending on what you want to run.

1. algo_eval (PyTorch / timm)

conda create -n timm-sfu python=3.8 pytorch torchvision torchaudio cudatoolkit -c pytorch
conda activate timm-sfu
conda install numpy scipy matplotlib ipykernel tqdm pandas dill
pip install scipy==1.8.1 statsmodels
pip install git+https://github.com/rwightman/pytorch-image-models.git
pip install torchpwl piecewise_regression tensorboardX

The pinned requirements.txt (in the repo root) lists the exact versions used in CI (torch==2.1.1, timm==0.9.11, piecewise-regression==1.2.1, transformers==4.35.0.dev0).

2. Hardware (Verilator / CocoTB)

cocoTB 1.6.2 only supports verilator 4.106:

git clone https://github.com/verilator/verilator
cd verilator
git checkout v4.106

Then edit include/verilatedos.h and change VL_MULS_MAX_WORDS from 16 to 128, and build:

./configure
make -j `nproc`
sudo make install

---

Sweeping every activation on SLURM

slurm/run_all.sh enumerates algo_eval/configs/, generates one *.slurm script per config, and submits it. Each job runs:

python -m algo_eval.pwl_eval2 --config ${FILE_woPY}

Edit the SLURM headers (#SBATCH …) and the conda env name at the top of run_all.sh to match your cluster.

---

Network-level evaluation

Once PWL models are trained and pickled, you can swap them into real networks:

• timm / ImageNet — python -m algo_eval.scripts.timm_validate … or algo_eval/pwl_eval_imagenet.py (loads pickles from algo_eval/src/pickles/).
• HuggingFace / GLUE — algo_eval/scripts/run_glue.py patches a transformer's activations with PWLs; collect_glue_accuracy.py aggregates the results.

---

SFU Hardware Eval

(See hw/ for the actual sources.)

5. Python virtual environment

python3 -m venv pysdk
source pysdk/bin/activate
pip install --upgrade pip
pip install -r requirements.txt