Awesome-GPU

• Architecture
  • Resources Management
  • Parallelism
  • Cache
  • Memory
  • White Papers
• Algorithms
  • BLAS
  • Stencils
  • Scans
• Applications
  • Deep Learning
• Tools
  • Benchmarks
  • Models
  • Simulators
  • Profilers
• Runtime
  • Scheduling
• Code Generation
  • Compilers
  • Programming Models
  • Profile Guided Optimization
  • Binaries

Architecture

Resources Management

• TECS'21-Reducing Energy in GPGPUs through Approximate Trivial Bypassing
• ASPLOS'17-Locality-Aware CTA Clustering for Modern GPUs
• ASPLOS'17-Dynamic Resource Management for Efficient Utilization of Multitasking GPUs
• HPCA'17-Dynamic GPGPU Power Management Using Adaptive Model Predictive Control
• ISCA'16-Transparent Offloading and Mapping (TOM): Enabling Programmer-Transparent Near-Data Processing in GPU Systems

Parallelism

• HPCA'18-Accelerate GPU Concurrent Kernel Execution by Mitigating Memory Pipeline Stalls
• HPCA'17-Controlled Kernel Launch for Dynamic Parallelism in GPUs
• GTC'17-COOPERATIVE GROUPS
• ISCA'16-LaPerm: Locality Aware Scheduler for Dynamic Parallelism on GPUs
• ISCA'16-Virtual Thread Maximizing Thread-Level Parallelism beyond GPU Scheduling Limit
• Berkeley TechRpts'16-Understanding Latency Hiding on GPUs

Cache

• ISCA'16-APRES: Improving Cache Efficiency by Exploiting Load Characteristics on GPUs
• SC'15-Adaptive and Transparent Cache Bypassing for GPUs

Memory

• ICCAD'21-Improving Inter-kernel Data Reuse With CTA-Page Coordination in GPGPU
• SC'21-In-Depth Analyses of Unified Virtual Memory System for GPU Accelerated Computing
• IBM'20-Umpire: Application-Focused Management and Coordination of Complex Hierarchical Memory
• HPCA'13-Reducing GPU Offload Latency via Fine-Grained CPU-GPU Synchronization

White Papers

• NVIDIA Hopper-NVIDIA H100 Tensor Core GPU Architecture
• NVIDIA Ampere-NVIDIA A100 Tensor Core GPU Architecture
• NVIDIA Turing-NVIDIA TURING GPU ARCHITECTURE
• NVIDIA Volta-NVIDIA TESLA V100
• NVIDIA Pascal-NVIDIA TESLA P100
• NVIDIA Kepler-NVIDIA’s Next Generation CUDA Compute Architecture: Kepler
• NVIDIA Fermi-NVIDIA’s Next Generation CUDA Compute Architecture: Fermi
• AMD CDNA 2-INTRODUCING AMD CDNA 2 ARCHITECTURE
• AMD CDNA-INTRODUCING AMD CDNA ARCHITECTURE

Algorithms

BLAS

• GTC'20-DEVELOPING CUDA KERNELS TO PUSH TENSOR CORES TO THE ABSOLUTE LIMIT ON NVIDIA A100
• IPDPS'20-Demystifying Tensor Cores to Optimize Half-Precision Matrix Multiply
• PPoPP'19-A Coordinated Tiling and Batching Framework for Efficient GEMM on GPU
• GTC'18-CUTLASS: CUDA TEMPLATE LIBRARY FOR DENSE LINEAR ALGEBRA AT ALL LEVELS AND SCALES

Stencils

• CGO'20-AN5D: Automated Stencil Framework for High-Degree Temporal Blocking on GPUs
• IPDPS'20-On Optimizing Complex Stencils on GPUs
• PPoPP'18-Register Optimizations for Stencils on GPUs

Scans

• NVResearch TechRpts'16-Single-pass Parallel Prefix Scan with Decoupled Look-back

Applications

Deep Learning

• PPoPP'21-Understanding and bridging the gaps in current GNN performance optimizations
• SC'21-E.T.: re-thinking self-attention for transformer models on GPUs
• OSDI'21-GNNAdvisor: An Adaptive and Efficient Runtime System for GNN Acceleration on GPUs
• SC'20-Sparse GPU Kernels for Deep Learning
• PPoPP'18-SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks
• HPCA'17-Towards Pervasive and User Satisfactory CNN across GPU Microarchitectures

Tools

Benchmarking

• GTC'18-Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking
• ISPASS'10-Demystifying GPU Microarchitecture through Microbenchmarking

Models

• PMBS'19-Instruction Roofline An insightful visual performance model for GPUs
• ECP'19-Performance Tuning of Scientific Codes with the Roofline Model
• GTC'18-VOLTA Architecture and performance optimization
• Synthesis Lectures on Computer Architecture'12-Performance Analysis and Tuning for General Purpose Graphics Processing Units (GPGPU)
• SC'10-Fundamental_Optimizations

Simulators

• ISPASS'10-Visualizing Complex Dynamics in Many-Core Accelerator Architectures
• ISPASS'09-Analyzing CUDA Workloads Using a Detailed GPU Simulator

Profilers

• PLDI'18-GPU Code Optimization using Abstract Kernel Emulation and Sensitivity Analysis
• CGO'18-CUDAAdvisor: LLVM-based runtime profiling for modern GPUs
• CCGRID'18-Exposing Hidden Performance Opportunities in High Performance GPU Applications
• THPC'16-Monitoring Heterogeneous Applications with the OpenMP Tools Interface
• Euro-Par'15-Identifying Optimization Opportunities Within Kernel Execution in GPU Codes
• SC'13-Effective sampling-driven performance tools for GPU-accelerated supercomputers
• ISPASS'12-Lynx: A dynamic instrumentation system for data-parallel applications on GPGPU architectures
• ICPP'11-Parallel Performance Measurement of Heterogeneous Parallel Systems with GPUs
• Vampir|Score-P
• TAU
• PAPI
• Allinea MAP
• Open|SpeedShop
• HPCToolkit
• Ingero - eBPF-based GPU causal observability agent. Traces CUDA Runtime/Driver APIs via uprobes and host kernel events to build causal chains explaining GPU latency. <2% overhead, production-safe.
• NVIDIA Nsight Systems
• NVIDIA Nsight Compute
• SASSI
• NVBit

Runtime

Scheduling

• PPoPP'22-CASE: A Compiler-Assisted SchEduling Framework for Multi-GPU Systems
• TPDS'20-cCUDA: Effective Co-Scheduling of Concurrent Kernels on GPUs

Code Generation

Compilers

• AMD'21-Generating GPU Compiler Heuristics using Reinforcement Learning
• TACO'21-Domain-Specific Multi-Level IR Rewriting for GPU: The Open Earth Compiler for GPU-accelerated Climate Simulation
• LLVM'17-Implementing implicit OpenMP data sharing on GPUs
• CGO'16-gpucc: An Open-Source GPGPU Compiler
• LLVM'16-Offloading Support for OpenMP in Clang and LLVM
• PMBS'15-Performance Analysis of OpenMP on a GPU using a CORAL Proxy Application
• LLVM'15-Integrating GPU Support for OpenMP Offloading Directives into Clang
• LLVM'14-Coordinating GPU Threads for OpenMP 4.0 in LLVM

Programming Models

• CGO'21-C-for-metal: high performance SIMD programming on intel GPUs
• ECRTS'19-Novel Methodologies for Predictable CPU-To-GPU Command Offloading
• ASPLOS'14-Paraprox: Pattern-Based Approximation for Data Parallel Applications

Profile Guided Optimization

• Geometry and Optimization'21-Cooperative Profile Guided Optimizations
• IPDPS'13-Kernel Specialization for Improved Adaptability and Performance on Graphics Processing Units (GPUs)

Binaries

• CGO'19-Decoding CUDA binary
• ISCA'15-Flexible software profiling of GPU architectures