[!WARNING] Elastic CI Stack for AWS v7 is now available. New features will only be developed and released for v7, and will not be backported to v6. Security fixes will continue to be backported to v6. See the v6 to v7 upgrade guide for details on migrating.

Buildkite Elastic CI Stack for AWS

[!NOTE] Prefer Terraform? See terraform-buildkite-elastic-ci-stack-for-aws

Buildkite provides a platform for running fast, secure, and scalable continuous integration pipelines on your own infrastructure.

The Buildkite Elastic CI Stack for AWS gives you a private, autoscaling Buildkite Agent cluster. Use it to parallelize large test suites across thousands of nodes, run tests and deployments for Linux or Windows based services and apps, or run AWS ops tasks.

Getting started

Learn more about the Elastic CI Stack for AWS and how to get started with it from the Buildkite Docs:

• Elastic CI Stack for AWS overview page, for a summary of the stack's architecture and supported features.
• Linux and Windows setup for the Elastic CI Stack for AWS page for a step-by-step guide on how to set up the Elastic CI Stack in AWS for these operating systems.

A list of recommended resources provides links to other pages in the Buildkite Docs for more detailed information.

Alternatively, jump straight in:

The current release is . See Releases for older releases.

Although the stack creates its own VPC by default, Buildkite highly recommends following best practices by setting up a separate development AWS account and using role switching and consolidated billing — see the Delegate Access Across AWS Accounts tutorial for more information.

If you want to use the AWS CLI, download config.json.example, rename it to config.json, update it with your agent token, then run the below AWS CLI command to create the AWS CloudFormation stack:

aws cloudformation create-stack \
  --output text \
  --stack-name buildkite \
  --template-url "https://s3.amazonaws.com/buildkite-aws-stack/latest/aws-stack.yml" \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
  --parameters "$(cat config.json)"

Security

This repository hasn't been reviewed by security researchers. Therefore, exercise caution and careful thought with what credentials you make available to your builds.

Anyone with commit access to your codebase (including third-party pull-requests if you've enabled them in Buildkite) will have access to your secrets bucket files.

Also, keep in mind the EC2 HTTP metadata server is available from within builds, which means builds act with the same IAM permissions as the instance.

Limiting CloudFormation Permissions

By default, CloudFormation will operate using the permissions granted to the identity of the credentials used to initiate a stack deployment or update.

If you want to explicitly specify which actions CloudFormation can perform on your behalf, you can either create your stack using credentials for an IAM identity with limited permissions, or provide an AWS CloudFormation service role.

🧑‍🔬 templates/service-role.yml template contains an experimental service role and set of IAM Policies that list the IAM Actions necessary to create, update, and delete a CloudFormation Stack created with the Buildkite Elastic CI Stack template. The role created by this template is currently being tested, but it has not been tested enough to be depended on. There are likely to be missing permissions for some stack parameter permutations.

aws cloudformation deploy --template-file templates/service-role.yml --stack-name buildkite-elastic-ci-stack-service-role --region us-east-1 --capabilities CAPABILITY_IAM

Experimental Resource Limits

The Elastic CI Stack includes configurable systemd resource limits to prevent resource exhaustion. These limits can be configured using CloudFormation parameters:

Example Configuration

To enable resource limits with custom values, include these parameters in your CloudFormation template or config file:

{
  "Parameters": {
    "ExperimentalEnableResourceLimits": "true",
    "ResourceLimitsMemoryHigh": "80%",
    "ResourceLimitsMemoryMax": "85%",
    "ResourceLimitsMemorySwapMax": "90%",
    "ResourceLimitsCPUWeight": "100",
    "ResourceLimitsCPUQuota": "85%",
    "ResourceLimitsIOWeight": "75"
  }
}

Notes

• Resource limits are disabled by default
• Values can be specified as percentages or absolute values (for memory-related parameters)

Agent watchdog

Buildkite marks agents that stop sending heartbeats as lost and fails their running jobs. On Linux, a watchdog powers off the instance after three minutes without an agent health notification. The Auto Scaling group replaces it, maintaining desired capacity.

Agent service failures and out-of-memory kills within the service also trigger instance replacement, interrupting all jobs on that instance. Normal graceful shutdown lets running jobs finish.

Scheduled Scaling

The Elastic CI Stack supports time-based scaling to automatically adjust the minimum number of instances based on your team's working hours. This feature helps optimize costs by scaling down during off-hours while allowing users the ability to proactively scale up capacity ahead of expected increasing capacity requirements.

Configuration Parameters

Example Configuration

To enable scheduled scaling that maintains a minimum of 10 ASG instances during business hours (8 AM - 6 PM, Eastern Time) and 2 ASG instances during off-hours:

{
  "Parameters": {
    "EnableScheduledScaling": "true",
    "ScheduleTimezone": "America/New_York",
    "ScaleUpSchedule": "0 8 * * MON-FRI",
    "ScaleUpMinSize": "10",
    "ScaleDownSchedule": "0 18 * * MON-FRI",
    "ScaleDownMinSize": "2"
  }
}

Schedule Format

Scheduled scaling uses AWS Auto Scaling cron expressions with the format:

minute hour day-of-month month day-of-week

Common examples:

• 0 8 * * MON-FRI - 8:00 AM on weekdays
• 0 18 * * MON-FRI - 6:00 PM on weekdays
• 0 9 * * SAT - 9:00 AM on Saturdays
• 30 7 * * 1-5 - 7:30 AM Monday through Friday (using numbers)

Timezone Support

The ScheduleTimezone parameter supports IANA timezone names such as:

• America/New_York (Eastern Time)
• America/Los_Angeles (Pacific Time)
• Europe/London (Greenwich Mean Time)
• Asia/Tokyo (Japan Standard Time)
• UTC (Coordinated Universal Time)

Development

To get started with customizing your own stack, or contributing fixes and features:

# Checkout all submodules
git submodule update --init --recursive

# Build all AMIs and render a cloud formation template - this requires AWS credentials (in the ENV)
# to build an AMI with packer
make build

# To create a new stack on AWS using the local template
make create-stack

# You can use any of the AWS* environment variables that the aws-cli supports
AWS_PROFILE="some-profile" make create-stack

# You can also use aws-vault or similar
aws-vault exec some-profile -- make create-stack

If you need to build your own AMIs (because you've changed something in the packer directory), run make packer with AWS credentials in your shell environment.

The build process is now two steps:

1. Base AMI: A base AMI is created with the latest OS updates. This image is cached and only rebuilt when the underlying OS or configuration changes.
2. Final AMI: The final AMI is built on top of the base AMI, installing the Buildkite agent and other software. This makes final builds faster and more consistent.

By default, AMIs are built as private. You can control AMI visibility and build location using these variables:

• AMI_PUBLIC - Set to true to make AMIs publicly accessible to all AWS users, or false (default) for private AMIs.
• AMI_USERS - A comma-separated list of AWS account IDs that should have access to private AMIs (ignored when AMI_PUBLIC=true).
• AWS_REGION - The AWS region where AMIs should be built (defaults to us-east-1).

# Build private AMIs (default - recommended for security)
make packer

# Build public AMIs (available to all AWS users)
make packer AMI_PUBLIC=true

# Build private AMIs with access for specific AWS accounts
make packer AMI_USERS="123456789012,987654321098,555666777888"

# Combined: private AMIs with specific account access in a different region
make packer AMI_PUBLIC=false AMI_USERS="123456789012,987654321098" AWS_REGION=us-west-2

This will create AMIs for the following platforms in your account's us-east-1 region (or the region specified by AWS_REGION):

• Linux (64-bit x86)
• Linux (64-bit Arm)
• Ubuntu (64-bit x86)
• Ubuntu (64-bit Arm)
• Windows (64-bit x86)

If you only need a subset, invoke the specific targets to skip the rest:

# Linux amd64 (x86_64)
make packer-base-linux-amd64.output packer-linux-amd64.output

# Linux arm64
make packer-base-linux-arm64.output packer-linux-arm64.output

# Both Linux architectures
make packer-base-linux-amd64.output packer-linux-amd64.output \
     packer-base-linux-arm64.output packer-linux-arm64.output

# Ubuntu amd64 (x86_64)
make packer-base-ubuntu2404-amd64.output packer-ubuntu2404-amd64.output

# Ubuntu arm64
make packer-base-ubuntu2404-arm64.output packer-ubuntu2404-arm64.output

Security Note: Making AMIs public (AMI_PUBLIC=true) can expose any secrets accidentally baked into the image. The default private setting helps prevent accidental exposure of sensitive information.

Release builds

Branch and PR builds use if_changed to select affected AMI builds and tests. Pushing a release tag builds and tests all final AMIs and publishes the CloudFormation template. Unchanged base AMIs reuse the cached builds from main.

Support Policy

We provide support for security and bug fixes on the current major release (v7) only. Security fixes will also be backported to v6. New features and non-security bug fixes will not be backported to v6.

If there are any changes in the main branch since the last tagged release, we aim to publish a new tagged release of this template at the end of each month.

AWS Regions

Buildkite:

• Supports all AWS Regions, except China and US GovCloud.
• Aims to support new regions within one month of general availability.

Operating Systems

Buildkite builds and deploys the following AMIs to all our supported regions:

• Amazon Linux 2023 (64-bit x86)
• Amazon Linux 2023 (64-bit Arm)
• Ubuntu 24.04 LTS (64-bit x86)
• Ubuntu 24.04 LTS (64-bit Arm)
• Windows Server 2022 (64-bit x86)

Amazon Linux 2023 is the default for Linux instances. To run Ubuntu 24.04 LTS instead, set the LinuxDistribution parameter to ubuntu2404.

Buildkite Agent

The Elastic CI Stack template published from the main branch tracks the latest Buildkite Agent release.

You may wish to preview any updates to your stack from this template using a CloudFormation Stack Change Set to decide whether to apply it. See the Elastic CI Stack v7 upgrade guide before upgrading from stack v6.

Recommended reading

Following on from the Getting started pages above, to gain a better understanding of how Elastic CI Stack works and how to use it most effectively and securely, see the following resources:

• Buildkite Agents in AWS overview
• Template parameters
• Using AWS Secrets Manager
• VPC design
• CloudFormation service role

Questions and support

Feel free to drop an email to [email protected] with questions. It'll also help us if you can provide the following details:

# List your stack parameters
aws cloudformation describe-stacks --stack-name MY_STACK_NAME \
  --query 'Stacks[].Parameters[].[ParameterKey,ParameterValue]' --output table

Collect logs from CloudWatch

Provide Buildkite with logs from CloudWatch Logs:

/buildkite/elastic-stack/{instance-id}
/buildkite/system/{instance-id}

Collect logs via script

utils/log-collector pulls your stack's CloudWatch logs into a tarball you can send to support.

# Lambda and ASG logs
./utils/log-collector -s my-stack-name

# Add a specific instance's logs too
./utils/log-collector -s my-stack-name -i i-0123456789abcdef0

You'll need the AWS CLI set up with credentials for the account the stack runs in. It only reads (CloudFormation, Lambda, CloudWatch Logs and Auto Scaling), so a read-only role is fine. Run it with -h for the other flags.

If a source has no logs you'll get a .EMPTY file instead of a .log, and a failed one leaves a .ERROR file. Whatever went wrong is also listed in COLLECTION-ERRORS.txt.

You can also visit our Forum and post a question in the Elastic CI Stack for AWS section!

Licence

See Licence.md (MIT)