Buildkite Agent Runtime

Buildkite Agent Runtime

The buildkite-agent binary provides subcommands for interacting with Buildkite from within running job steps — creating annotations, uploading artifacts, sharing state between jobs, generating dynamic pipelines, requesting OIDC tokens, and more. This skill covers the command syntax, flags, and patterns for every in-job subcommand.

Quick Start

A step that runs tests, annotates failures, uploads coverage, and stores a result flag for downstream jobs:

steps:
  - label: ":test_tube: Tests"
    command: |
      if ! make test 2>&1 | tee test-output.txt; then
        buildkite-agent annotate --style "error" --context "test-failures"  For pipeline-level `notify:` configuration, see the **buildkite-pipelines** skill.

## Artifacts

Upload files as build artifacts, download them in later steps or other builds, and search by glob.

### Upload

```bash
# Upload a single file
buildkite-agent artifact upload "pkg/release.tar.gz"

# Upload with glob pattern
buildkite-agent artifact upload "dist/**/*"

Download

# Download to current directory
buildkite-agent artifact download "pkg/release.tar.gz" .

# Download from a specific step
buildkite-agent artifact download "dist/*" . --step "build-step"

Search

# List matching artifacts
buildkite-agent artifact search "pkg/*.tar.gz" --build "$BUILDKITE_BUILD_ID"

For complete flag tables, see references/flag-reference.md.

> For the declarative artifact_paths: YAML key, see the buildkite-pipelines skill. For bk artifact CLI commands, see the buildkite-cli skill.

Meta-data

A build-wide key-value store for sharing state between jobs. Set a value in one job, read it in any other job in the same build.

Set

buildkite-agent meta-data set "release-version" "1.4.2"

Get

VERSION=$(buildkite-agent meta-data get "release-version")

Use --default to return a fallback value instead of a non-zero exit when the key is missing: buildkite-agent meta-data get "deploy-env" --default "staging".

Check existence

# Returns exit code 0 if exists, 100 if not
if buildkite-agent meta-data exists "release-version"; then
  echo "Version already set"
fi

Common patterns

Block step field values are stored automatically as meta-data. Retrieve them by field key:

# After a block step with fields: [{key: "release-name", text: "Release Name"}]
RELEASE_NAME=$(buildkite-agent meta-data get "release-name")

Pipeline Upload

Dynamically add steps to a running build. The core mechanism behind dynamic pipelines — generate YAML at runtime and upload it.

Basic usage

# Upload a specific file
buildkite-agent pipeline upload .buildkite/deploy-steps.yml

# Pipe generated YAML from stdin
./scripts/generate-pipeline.sh | buildkite-agent pipeline upload

Replace mode

By default, uploaded steps are appended after the current step. Use --replace to replace the entire remaining pipeline:

# Replace all remaining steps with the uploaded ones
buildkite-agent pipeline upload --replace .buildkite/new-pipeline.yml

Key flags

| Flag | Default | Description | |------|---------|-------------| | --replace | false | Replace remaining pipeline steps instead of appending | | --no-interpolation | false | Skip environment variable interpolation in the uploaded YAML | | --dry-run | false | Validate and output the pipeline without uploading |

> For pipeline YAML syntax and step types, see the buildkite-pipelines skill.

OIDC Tokens

Request short-lived OpenID Connect tokens from within a job step for authenticating to external services (cloud providers, package registries) without static credentials.

Basic token request

# Request a token for a specific audience
TOKEN=$(buildkite-agent oidc request-token --audience "https://packages.buildkite.com/my-org/my-registry")

Cloud provider authentication

# AWS — request token with STS audience
TOKEN=$(buildkite-agent oidc request-token --audience "sts.amazonaws.com")

Key flags

| Flag | Default | Description | |------|---------|-------------| | --audience | Buildkite endpoint | Target service URL — must match the OIDC provider audience configuration | | --lifetime | 0 (API default) | Token lifetime in seconds. When 0 or omitted, the API chooses a default lifetime | | --claim | — | Optional claims to include (e.g., organization_id,pipeline_id). Repeatable | | --aws-session-tag | — | Claims to map as AWS session tags. Repeatable |

> For end-to-end OIDC auth flows, cloud provider setup, and token claim details, see the buildkite-secure-delivery skill.

Step Management

Read or modify step attributes at runtime. Useful for conditional logic within steps and build automation.

Get step attributes

# Get a step's label (--step is required)
LABEL=$(buildkite-agent step get "label" --step "$BUILDKITE_STEP_KEY")

# Get another step's state
STATE=$(buildkite-agent step get "state" --step "deploy-step")

# Get the entire step as JSON
buildkite-agent step get --step "test-step" --format json

Update step attributes

# Update a step's label dynamically
buildkite-agent step update "label" ":rocket: Deploying v${VERSION}" --step "$BUILDKITE_STEP_KEY"

# Append to an existing label
buildkite-agent step update "label" " (retried)" --step "build-step" --append

Cancel a step

# Cancel all unfinished jobs for a step
buildkite-agent step cancel --step "optional-step"

# Force-cancel a running step
buildkite-agent step cancel --step "stuck-step" --force

Key flags

| Flag | Default | Description | |------|---------|-------------| | --step | — | Step key or UUID to target (required — uses $BUILDKITE_STEP_ID env var if set) | | --build | current build | Build UUID (for cross-build operations) | | --format | — | Output format for get (use json for complex attributes) | | --append | false | Append to existing value instead of replacing (step update only) | | --force | false | Force cancel even if the step is running (step cancel only) |

Distributed Locks

Coordinate parallel jobs within a build using distributed mutex locks. Prevents race conditions when multiple jobs access shared resources.

Acquire / release pattern

#!/bin/bash
set -euo pipefail

# Acquire lock — blocks until available, returns a token
token=$(buildkite-agent lock acquire "database-migration")
trap 'buildkite-agent lock release "database-migration" "${token}"' EXIT

# Critical section — only one job runs this at a time
bundle exec rails db:migrate

Do / done pattern (one-time setup)

Run a setup task exactly once across all parallel jobs:

#!/bin/bash
echo "+++ Setting up shared test environment"

if [[ $(buildkite-agent lock do "test-env-setup") == "do" ]]; then
  echo "Downloading test assets..."
  curl -o /tmp/test-data.zip https://releases.example.com/data.zip
  unzip /tmp/test-data.zip -d /tmp/shared-test-files/
  buildkite-agent lock done "test-env-setup"
else
  echo "Assets already prepared by another job"
fi

# All jobs continue here
run-tests.sh

Key flags

| Subcommand | Flags | Description | |------------|-------|-------------| | lock acquire | --lock-wait-timeout | Maximum wait duration (e.g. 30s, 5m). Default: wait forever | | lock release | — | Release with the token from acquire | | lock do | — | Returns do if lock acquired, done if already completed | | lock done | — | Mark a do lock as completed |

Environment

Inspect and modify the job's environment variables. Primarily useful for debugging lifecycle hooks and understanding what environment changes hooks made.

# Dump all environment variables as JSON
buildkite-agent env dump | jq .

# Get a specific variable
buildkite-agent env get "BUILDKITE_BRANCH"

# Set variables for subsequent hooks and the command (KEY=value format)
buildkite-agent env set DEPLOY_TARGET=production "APP_NAME=My App"

Key flags

| Subcommand | Description | |------------|-------------| | env dump | Dump all environment variables (JSON format by default) | | env get | Get one or more specific variables. Use --format for output format (plain, json, json-pretty) | | env set KEY=value [...] | Set variables for subsequent phases. Accepts multiple KEY=value pairs | | env unset | Remove a variable from subsequent phases |

Debugging hooks

The env dump command is particularly useful in lifecycle hooks to see what prior hooks changed:

#!/bin/bash
# .buildkite/hooks/pre-command
echo "--- Environment after environment hook:"
buildkite-agent env dump | jq 'keys'

> For agent lifecycle hooks and buildkite-agent.cfg configuration, see the buildkite-agent-infrastructure skill.

Secrets

Retrieve cluster secrets at runtime from within job steps. Secrets retrieved this way are automatically added to the log redactor.

Basic usage

# Get a secret value
SECRET_VAR=$(buildkite-agent secret get "deploy-key")

# Pass directly to a tool
cli-tool --token "$(buildkite-agent secret get "api-token")"

Key flags

| Flag | Default | Description | |------|---------|-------------| | --format | default | Output format: default (single secret prints value, multiple prints JSON), json, or env (KEY="value" pairs) | | --skip-redaction | false | Do not add the secret value to the log redactor | | --job | current job | Job ID context |

Multiple secret keys can be requested at once: buildkite-agent secret get KEY1 KEY2 KEY3.

By default, secret get automatically registers retrieved values with the log redactor, masking them as [REDACTED] in subsequent output.

> For setting up cluster secrets, see the buildkite-agent-infrastructure skill. For the declarative secrets: pipeline YAML key, see the buildkite-pipelines skill.

Log Redaction

Add values to the build log redactor at runtime so they are masked in all subsequent output. Use this for dynamically-retrieved secrets that were not declared via secrets: or buildkite-agent secret get.

Basic usage

# Fetch a token from an external source
DYNAMIC_TOKEN=$(curl -s https://vault.example.com/token)

# Register it with the redactor before using it
echo "$DYNAMIC_TOKEN" | buildkite-agent redactor add

# Now any log output containing the token value shows [REDACTED]
echo "Using token: $DYNAMIC_TOKEN"
# Output: Using token: [REDACTED]

Multiple values

# Redact multiple values
echo "$SECRET1" | buildkite-agent redactor add
echo "$SECRET2" | buildkite-agent redactor add

When to use redactor vs secret get

| Scenario | Use | |----------|-----| | Secret stored in Buildkite cluster secrets | buildkite-agent secret get (auto-redacts) | | Secret from external vault (HashiCorp Vault, AWS SSM, etc.) | Fetch externally, then buildkite-agent redactor add | | Computed sensitive value (temporary token, derived key) | buildkite-agent redactor add |

Tool Signing

Sign pipeline YAML so that agents can verify step integrity before execution. The tool sign command takes a pipeline file (not individual steps) and annotates it with signatures.

Sign a pipeline from a file

# Sign a pipeline YAML file using a local JWKS key
buildkite-agent tool sign pipeline.yml \
  --jwks-file /path/to/private-key.json \
  --repo "git@github.com:org/repo.git"

Sign via the GraphQL API

# Retrieve, sign, and update a pipeline via the Buildkite GraphQL API
buildkite-agent tool sign \
  --graphql-token "$BUILDKITE_GRAPHQL_TOKEN" \
  --organization-slug my-org \
  --pipeline-slug my-pipeline \
  --jwks-file /path/to/private-key.json \
  --update

Generate a signing key pair

# Generate a new JWS key pair (private + public JWKS files)
buildkite-agent tool keygen

Key flags

| Command | Flag | Description | |---------|------|-------------| | tool sign | --jwks-file | Path to JWKS private key file for signing | | tool sign | --jwks-key-id | Key ID to use from the JWKS file | | tool sign | --repo | Repository URL (required when signing from a file) | | tool sign | --signing-aws-kms-key | AWS KMS key ID for signing (alternative to JWKS) | | tool sign | --signing-gcp-kms-key | GCP KMS key ID for signing (alternative to JWKS) | | tool sign | --graphql-token | Token for retrieving/updating pipeline via GraphQL API | | tool sign | --update | Update the pipeline in Buildkite after signing (requires --graphql-token) | | tool keygen | --alg | JWS signing algorithm (default: EdDSA) | | tool keygen | --key-id | Key ID for the generated pair (default: random) |

> Note: There is no tool verify command. Signature verification is handled internally by the agent when it receives a job.

> For pipeline signing configuration and rollout strategy, see the buildkite-secure-delivery skill.

Common Mistakes

| Mistake | What happens | Fix | |---------|-------------|-----| | Missing --context on annotate | Each call creates a new annotation instead of updating | Always pass --context with a stable identifier | | Using --append without matching --context | Append has no effect — creates a new annotation | Ensure --context matches the annotation to append to | | Forgetting to quote artifact glob patterns | Shell expands globs before buildkite-agent sees them | Always quote: "dist/**/*" not dist/**/* | | Reading meta-data get before the writing job completes | Key does not exist, command fails with non-zero exit | Use depends_on or wait to enforce ordering, or use --default | | Using pipeline upload --replace unintentionally | Removes all remaining steps in the build | Only use --replace when intentionally rebuilding the entire pipeline | | Not releasing locks on script failure | Lock held indefinitely, blocking other jobs | Use trap ... EXIT to release locks on any exit | | Passing --audience that doesn't match OIDC provider config | Token rejected by the target service | Audience must exactly match the provider's configured audience URL | | Using --skip-redaction with actual secrets | Secret values appear in plain text in build logs | Only use --skip-redaction for non-sensitive configuration values | | Calling env set expecting it to affect the current shell | Variable is set for subsequent hooks/phases, not the current script | Use export VAR=value for current-script variables; env set for cross-phase | | Passing large values via environment variables | OS-level env size limits cause silent truncation or job failure | Switch to file-based approaches (artifacts, meta-data with files) for payloads larger than a few KB | | Uploading pipeline YAML with unescaped $ in --no-interpolation mode off | Variables interpolated unexpectedly, producing malformed YAML | Use --no-interpolation when YAML contains literal $ characters |

Additional Resources

Reference Files

• references/flag-reference.md — Complete flag tables for all subcommands including upload, download, search, shasum, annotate, meta-data, pipeline upload, oidc, step, lock, env, secret, redactor, and tool
• references/patterns-and-recipes.md — Advanced multi-subcommand patterns: test failure annotation pipelines, cross-job state machines, OIDC-authenticated Docker push, parallel job coordination with locks, environment debugging

Further Reading

• Buildkite Docs for LLMs
• Agent CLI — annotate
• Agent CLI — artifact
• Agent CLI — meta-data
• [Agent CLI — pipeline](https://buildkite.com/docs/agent/v3/cli-pipeline.md

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: buildkite
• Source: buildkite/skills
• License: MIT

Install and usage instructions live in the source repository linked above.