Ci Pipelines

CI Pipelines

CI in open source has a constraint that internal CI does not: most PRs come from forks, by people who cannot debug your pipeline and will not wait for it. Design for that.

Targets

| Metric | Target | Why | |---|---|---| | PR feedback time | < 10 min | Beyond this, contributors context-switch away | | Lint/format feedback | < 2 min | Fail fast on the cheap stuff | | Flake rate | < 1% | Above this, red CI gets ignored (see testing-strategy) | | Fork PR success | 100% of non-secret jobs | A contributor must be able to get green |

The last one is the one projects get wrong. If a fork PR always shows a red X because a deploy job cannot access secrets, every contributor's first experience is failure.

Workflow structure

Split by purpose and speed, not into one giant workflow:

.github/workflows/
├── ci.yml            # lint + test on PR and push to main — the required check
├── release.yml       # tag-triggered publish (see release-engineering)
├── codeql.yml        # scheduled security scanning
├── nightly.yml       # slow: full matrix, fuzzing, benchmarks, downstream tests
└── docs.yml          # deploy docs on merge to main

A minimal, correct ci.yml:

name: CI
on:
  push:
    branches: [main]
  pull_request:

# Cancel superseded runs — the single biggest CI cost saving, one block long
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

permissions:
  contents: read          # least privilege by default; grant per-job as needed

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '22', cache: 'npm' }
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck

  test:
    needs: lint                      # don't burn matrix minutes on unformatted code
    strategy:
      fail-fast: false               # show every failing cell, not just the first
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node: ['20', '22']
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: ${{ matrix.node }}, cache: 'npm' }
      - run: npm ci
      - run: npm test

Details that matter and are usually missing:

• concurrency with cancel-in-progress — a contributor pushing five times in ten

minutes should not queue five full matrices.

• fail-fast: false — otherwise one Windows failure hides three others.
• permissions: contents: read at the top level, elevated only in the job that

needs it. Default-permissive tokens are the main blast radius in a compromised action.

• needs: lint — cheap gate first.
• Pin third-party actions to a SHA, not a tag (see supply-chain-security).

First-party actions/* by major tag is a defensible compromise; anything else, pin.

Matrix sizing

A full cartesian product is the default and it is almost always wasteful. 3 OS × 4 versions × 2 configs = 24 jobs for a project whose bugs are all on one axis.

Test the corners exhaustively and the middle sparsely:

strategy:
  fail-fast: false
  matrix:
    os: [ubuntu-latest]
    python: ['3.10', '3.13']            # oldest supported + newest
    include:
      - { os: macos-latest,   python: '3.13' }
      - { os: windows-latest, python: '3.13' }
      - { os: ubuntu-latest,  python: '3.12' }

Move the exhaustive matrix to nightly.yml. PRs get fast signal; the full grid still runs daily and you learn about the rare cell within 24 hours.

Caching

Cache the dependency store, never node_modules/.venv themselves — restoring a partially valid install directory produces confusing failures.

- uses: actions/cache@v4
  with:
    path: ~/.cargo/registry
    key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
    restore-keys: ${{ runner.os }}-cargo-

Rules:

• Key on the lockfile hash. Keying on anything vaguer serves stale caches.
• Include the OS and language version in the key, or macOS restores Linux binaries.
• restore-keys for partial hits — a near-miss cache still saves most of the time.
• Language-specific setup actions already do this (cache: 'npm', cache: 'pip').

Use them before hand-rolling.

• Never cache anything that affects correctness — build outputs keyed loosely

produce "works in CI, broken for users", the worst failure mode available.

• GitHub evicts caches after 7 days of no use, 10 GB per repo. Do not architect

around a cache being present.

Fork PRs and secrets

The core security constraint: pull_request from a fork gets a read-only token and no secrets. This is correct and you should not fight it.

• pull_request_target runs with full secrets against the base repo and is the

most dangerous trigger in GitHub Actions. If you use it, never check out the PR head — that executes attacker-controlled code with your secrets. Use it only for labeling, commenting, and other metadata work.

• Structure jobs so fork PRs go fully green without secrets. Guard secret-requiring

jobs: ``yaml if: github.event.pull_request.head.repo.full_name == github.repository ``

• Coverage upload, preview deploys, and publishing all belong in a separate

workflow triggered by workflow_run after CI succeeds, or on push to main.

• Require approval for first-time contributors' workflow runs — repo Settings →

Actions. This is the default for new contributors and worth keeping.

• Self-hosted runners on a public repo are a remote code execution service unless

they are ephemeral and isolated. Every fork PR can run arbitrary code on them. If you need self-hosted, use ephemeral single-use runners in a disposable VM.

Required checks and branch protection

Configure on main:

• Require status checks to pass, and require branches be up to date only if your

merge volume is low — on a busy repo that setting causes an update-rebase treadmill. Prefer a merge queue.

• Require one approving review, dismiss stale approvals on new commits.
• Require conversation resolution.
• Include administrators. If you exempt yourself, the rules are advisory.
• Do not require jobs that cannot run on forks — this is the most common way a

repo becomes unmergeable for outside contributors.

Name required checks stably. Renaming a job silently makes the old required check unsatisfiable and blocks every PR until someone notices.

Keeping CI fast

In order of impact:

1. Cancel superseded runs (concurrency). Often halves total minutes.
2. Fail fast on cheap jobs. Lint before matrix.
3. Cache dependencies. Usually the single largest per-job cost.
4. Shrink the PR matrix, move the rest to nightly.
5. Parallelize the test suite (pytest -n auto, cargo nextest, sharded jest).
6. Only run what changed — paths-filter for monorepos, or a task runner with a

dependency graph (Nx, Turborepo, Bazel).

1. Skip docs-only changes:

``yaml on: pull_request: paths-ignore: ['**.md', 'docs/**'] ` Careful: if the workflow is a required check, paths-ignore` makes it never run and never satisfy the requirement. Use a "skip job that reports success" pattern instead.

Public repos get free standard-runner minutes on GitHub-hosted runners; private repos do not. Either way, a 40-minute pipeline costs contributor attention, which is the scarcer resource.

Making CI failures debuggable by strangers

A contributor who cannot understand the failure will abandon the PR.

• Name jobs for what they check: test (ubuntu, py3.13), not build-2.
• Print the reproduction command in the failure path: Run \make test-unit\` to

reproduce locally.`

• Upload artifacts on failure — logs, screenshots, core dumps:

```yaml

• uses: actions/upload-artifact@v4

if: failure() with: { name: logs-${{ matrix.os }}, path: ./logs } ```

• Use annotations so errors appear inline on the diff (most linters have a GitHub

reporter; reviewdog wraps the rest).

• Document the flaky jobs you know about and how to tell them apart from real

failures. Better: fix them.

Anti-patterns

• No concurrency block. Wasted minutes and contributor waiting time.
• pull_request_target + checkout of PR head. Full compromise; treat as critical.
• Secrets in a job that fork PRs run. They won't work, and the red X is on the

contributor.

• Required checks that forks cannot satisfy.
• Unpinned third-party actions. @v1 is a moving target someone else controls.
• A 30-job matrix on every PR.
• Caching build output keyed on the branch name.
• Auto-merge on green with no review. CI checks what you thought to check.
• Self-hosted runners on public repos, non-ephemeral.
• Muting a flaky job instead of fixing it. You just disabled the test suite.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: the-open-agent
• Source: the-open-agent/oss-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.