A curated list of resources for learning and exploring Triton, OpenAI's programming language for writing efficient GPU code.
Python
501
83 commits
updated Mar 10, 2025
A curated list of resources for learning and exploring Triton, OpenAI's programming language for writing efficient GPU code.
This project is a step-by-step learning journey where we implement various types of Triton kernels—from the simplest examples to more advanced applications—while exploring GPU programming with Triton. The goal of this repository is to help you (and others) get comfortable with Triton by:
Daily challenges: every day, a new challenge is posted in this repository. Each challenge focuses on a specific aspect of Triton, such as:
| Day | Kernel | Description |
|---|---|---|
| #1 | Constant add | This challenge is the first puzzle in our Daily Triton Challenge series. The goal is to write a Triton kernel that adds a constant value to each element of a vector. |
| #2 | Add two vectors | Simple example of how to add two vectors using a custom GPU kernel written in Triton and compares the result to a standard PyTorch implementation. |
| #3 | Add two vectors with speed benchmarking | This is almost the same as #2 but we meaesure kernel execution speed and compare it to Pytorch implementation. |
| #4 | ReLU Activation with Triton | In this challenge, you will implement the ReLU (Rectified Linear Unit) activation function using Triton. ReLU is defined as: ReLU(x)=max(0,x) |
| #5 | ReLU Activation forward and backward | In this challenge, you will implement the ReLU activation function in a way that is fully compatible with PyTorch’s autograd. That means you’ll write a custom autograd function that uses a Triton kernel for the forward pass (computing y = max(0, x)) and a second Triton kernel for the backward pass (computing the gradient of ReLU, where grad_input = grad_output if x > 0 and 0 otherwise). |
Gain deeper insights into Triton through these detailed articles:
Explore the academic foundation of Triton:
Learn by watching these informative videos:
Watch Triton community meetups to be up to date with Triton recent topics.
Challenge yourself with these engaging puzzles:
Enhance your Triton development workflow with these tools:
Catch up on the latest advancements from Triton Conferences:
Explore practical implementations with these sample kernels:
| Kernel | Description | Resource |
|---|---|---|
| VectorAdd | A simple kernel that performs element-wise addition of two vectors. Useful for understanding the basics of GPU programming in Triton. | 1 2 |
| Matmul | An optimized kernel for matrix multiplication, achieving high performance by leveraging memory hierarchy and parallelism. | 1 2 Grouped GEMM |
| Softmax | A kernel for efficient computation of the softmax function, commonly used in machine learning models like transformers. | 1 2 3 |
| Dropout | A kernel for implementing low-memory dropout, a regularization technique to prevent overfitting in neural networks. | 1 2 |
| Layer Normalization | A kernel for layer normalization, which normalizes activations within a layer to improve training stability in deep learning models. | 1 2 3 |
| Fused Attention | A kernel that efficiently implements attention mechanisms by combining multiple operations, key to transformers and similar architectures. | 1 2 |
| Conv1d | A kernel for 1D convolution, often used in processing sequential data like time series or audio signals. | 1 |
| Conv2d | A kernel for 2D convolution, a fundamental operation in computer vision tasks such as image classification or object detection. | 1 |
| MultiheadAttention | A kernel for multi-head attention, a crucial component in transformer-based models for capturing complex relationships in data. | 1 |
| Hardsigmoid | A kernel for the Hardsigmoid activation function, an efficient approximation of the sigmoid function used in certain neural network layers. | 1 |
| GeLU | GeLU | 1 |
| GeGLU | GeGLU | 1 |
| RMSNorm | RMSNorm | 1 |
Feel free to contribute more resources or suggest updates by opening a pull request or issue in this repository.
This resource list is open-sourced under the MIT license.
82 followers · starred Jul 2025
425 followers · starred Dec 2025
161 followers · starred Jul 2025
146 followers · starred Dec 2024
A curated list of resources for learning and exploring Triton, OpenAI's programming language for writing efficient GPU code.
Python
501
83 commits
updated Mar 10, 2025
A curated list of resources for learning and exploring Triton, OpenAI's programming language for writing efficient GPU code.
This project is a step-by-step learning journey where we implement various types of Triton kernels—from the simplest examples to more advanced applications—while exploring GPU programming with Triton. The goal of this repository is to help you (and others) get comfortable with Triton by:
Daily challenges: every day, a new challenge is posted in this repository. Each challenge focuses on a specific aspect of Triton, such as:
| Day | Kernel | Description |
|---|---|---|
| #1 | Constant add | This challenge is the first puzzle in our Daily Triton Challenge series. The goal is to write a Triton kernel that adds a constant value to each element of a vector. |
| #2 | Add two vectors | Simple example of how to add two vectors using a custom GPU kernel written in Triton and compares the result to a standard PyTorch implementation. |
| #3 | Add two vectors with speed benchmarking | This is almost the same as #2 but we meaesure kernel execution speed and compare it to Pytorch implementation. |
| #4 | ReLU Activation with Triton | In this challenge, you will implement the ReLU (Rectified Linear Unit) activation function using Triton. ReLU is defined as: ReLU(x)=max(0,x) |
| #5 | ReLU Activation forward and backward | In this challenge, you will implement the ReLU activation function in a way that is fully compatible with PyTorch’s autograd. That means you’ll write a custom autograd function that uses a Triton kernel for the forward pass (computing y = max(0, x)) and a second Triton kernel for the backward pass (computing the gradient of ReLU, where grad_input = grad_output if x > 0 and 0 otherwise). |
Gain deeper insights into Triton through these detailed articles:
Explore the academic foundation of Triton:
Learn by watching these informative videos:
Watch Triton community meetups to be up to date with Triton recent topics.
Challenge yourself with these engaging puzzles:
Enhance your Triton development workflow with these tools:
Catch up on the latest advancements from Triton Conferences:
Explore practical implementations with these sample kernels:
| Kernel | Description | Resource |
|---|---|---|
| VectorAdd | A simple kernel that performs element-wise addition of two vectors. Useful for understanding the basics of GPU programming in Triton. | 1 2 |
| Matmul | An optimized kernel for matrix multiplication, achieving high performance by leveraging memory hierarchy and parallelism. | 1 2 Grouped GEMM |
| Softmax | A kernel for efficient computation of the softmax function, commonly used in machine learning models like transformers. | 1 2 3 |
| Dropout | A kernel for implementing low-memory dropout, a regularization technique to prevent overfitting in neural networks. | 1 2 |
| Layer Normalization | A kernel for layer normalization, which normalizes activations within a layer to improve training stability in deep learning models. | 1 2 3 |
| Fused Attention | A kernel that efficiently implements attention mechanisms by combining multiple operations, key to transformers and similar architectures. | 1 2 |
| Conv1d | A kernel for 1D convolution, often used in processing sequential data like time series or audio signals. | 1 |
| Conv2d | A kernel for 2D convolution, a fundamental operation in computer vision tasks such as image classification or object detection. | 1 |
| MultiheadAttention | A kernel for multi-head attention, a crucial component in transformer-based models for capturing complex relationships in data. | 1 |
| Hardsigmoid | A kernel for the Hardsigmoid activation function, an efficient approximation of the sigmoid function used in certain neural network layers. | 1 |
| GeLU | GeLU | 1 |
| GeGLU | GeGLU | 1 |
| RMSNorm | RMSNorm | 1 |
Feel free to contribute more resources or suggest updates by opening a pull request or issue in this repository.
This resource list is open-sourced under the MIT license.
82 followers · starred Jul 2025
425 followers · starred Dec 2025
161 followers · starred Jul 2025
146 followers · starred Dec 2024
