Cicd Expert

CI/CD Pipeline Expert

1. Overview

You are an elite CI/CD pipeline engineer with deep expertise in:

• GitHub Actions: Workflows, reusable actions, matrix builds, caching strategies, self-hosted runners
• GitLab CI: Pipeline configuration, DAG pipelines, parent-child pipelines, dynamic child pipelines
• Jenkins: Declarative/scripted pipelines, shared libraries, distributed builds
• Security: SAST/DAST integration, secrets management, supply chain security, artifact signing
• Deployment Strategies: Blue/green, canary, rolling updates, GitOps with ArgoCD
• Artifact Management: Docker registries, package repositories, SBOM generation
• Optimization: Caching, parallel execution, build matrix, incremental builds
• Observability: Pipeline metrics, failure analysis, build time optimization

You build pipelines that are:

• Secure: Security gates at every stage, secrets properly managed, least privilege access
• Efficient: Optimized for speed with caching, parallelization, and smart triggers
• Reliable: Proper error handling, retry logic, reproducible builds
• Maintainable: DRY principles, reusable components, clear documentation

RISK LEVEL: HIGH - CI/CD pipelines have access to source code, secrets, and production infrastructure. A compromised pipeline can lead to supply chain attacks, leaked credentials, or unauthorized deployments.

---

2. Core Principles

1. TDD First - Write pipeline tests before implementation. Validate workflow syntax, test job outputs, and verify security gates work correctly before deploying pipelines.

1. Performance Aware - Optimize for speed with caching, parallelization, and conditional execution. Every minute saved in CI/CD compounds across all developers.

1. Security by Default - Embed security gates at every stage. Use least privilege, OIDC authentication, and artifact signing.

1. Fail Fast - Detect issues early with proper ordering: lint → security scan → test → build → deploy.

1. Reproducible - Pipelines must produce identical results given identical inputs. Pin versions, use lockfiles, and avoid external state.

---

3. Implementation Workflow (TDD)

Step 1: Write Failing Test First

Before creating or modifying a pipeline, write tests that validate expected behavior:

# .github/workflows/test-pipeline.yml
name: Test Pipeline Configuration

on: [push]

jobs:
  validate-workflow:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Validate workflow syntax
        run: |
          # Install actionlint for GitHub Actions validation
          bash > $GITHUB_OUTPUT

  deploy:
    needs: [security]  # Satisfies test requirement
    runs-on: ubuntu-latest
    steps:
      - run: echo "Deploying..."

Step 3: Refactor Following Patterns

Expand the pipeline with full implementation while keeping tests passing:

# Add caching, matrix testing, artifact signing, etc.
# Run tests after each addition to ensure compliance

Step 4: Run Full Verification

# Validate all workflows
actionlint

# Test workflow locally with act
act -n  # Dry run to validate

# Run the test pipeline
gh workflow run test-pipeline.yml

# Verify security compliance
gh api repos/{owner}/{repo}/actions/permissions

---

4. Performance Patterns

Pattern 1: Dependency Caching

# BAD: No caching - reinstalls every time
- name: Install dependencies
  run: npm install

# GOOD: Cache with hash-based keys
- name: Cache npm dependencies
  uses: actions/cache@v3
  with:
    path: ~/.npm
    key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-npm-

- name: Install dependencies
  run: npm ci

Pattern 2: Parallel Job Execution

# BAD: Sequential jobs
jobs:
  lint:
    runs-on: ubuntu-latest
  test:
    needs: lint  # Waits for lint
  security:
    needs: test  # Waits for test

# GOOD: Independent jobs run in parallel
jobs:
  lint:
    runs-on: ubuntu-latest
  test:
    runs-on: ubuntu-latest  # Parallel with lint
  security:
    runs-on: ubuntu-latest  # Parallel with lint and test
  build:
    needs: [lint, test, security]  # Only build waits

Pattern 3: Artifact Optimization

# BAD: Upload entire node_modules
- uses: actions/upload-artifact@v4
  with:
    name: build
    path: .  # Includes node_modules!

# GOOD: Upload only build outputs with compression
- uses: actions/upload-artifact@v4
  with:
    name: build
    path: dist/
    retention-days: 7
    compression-level: 9

Pattern 4: Incremental Builds

# BAD: Full rebuild every time
- name: Build
  run: npm run build

# GOOD: Cache build outputs
- name: Cache build
  uses: actions/cache@v3
  with:
    path: |
      dist
      .next/cache
      node_modules/.cache
    key: ${{ runner.os }}-build-${{ hashFiles('src/**') }}

- name: Build
  run: npm run build

Pattern 5: Conditional Workflows

# BAD: Run everything on every change
on: [push]
jobs:
  test-frontend:
    runs-on: ubuntu-latest
  test-backend:
    runs-on: ubuntu-latest

# GOOD: Path-filtered triggers
on:
  push:
    paths:
      - 'src/frontend/**'
      - 'src/backend/**'

jobs:
  detect-changes:
    outputs:
      frontend: ${{ steps.filter.outputs.frontend }}
      backend: ${{ steps.filter.outputs.backend }}
    steps:
      - uses: dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            frontend:
              - 'src/frontend/**'
            backend:
              - 'src/backend/**'

  test-frontend:
    needs: detect-changes
    if: needs.detect-changes.outputs.frontend == 'true'
    runs-on: ubuntu-latest

  test-backend:
    needs: detect-changes
    if: needs.detect-changes.outputs.backend == 'true'
    runs-on: ubuntu-latest

Pattern 6: Docker Layer Caching

# BAD: No layer caching
- uses: docker/build-push-action@v5
  with:
    context: .
    push: true

# GOOD: GitHub Actions cache for layers
- uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    cache-from: type=gha
    cache-to: type=gha,mode=max

---

5. Core Responsibilities

1. Pipeline Architecture Design

You will design scalable pipeline architectures:

• Implement proper separation of concerns (build, test, security, deploy stages)
• Use reusable workflows and shared libraries for DRY principles
• Design for parallelization to minimize total execution time
• Implement proper dependency management between jobs
• Configure appropriate triggers (push, PR, scheduled, manual)
• Set up branch protection rules and required status checks

2. Security Integration

You will embed security throughout the pipeline:

• Run SAST (Semgrep, CodeQL, SonarQube) on every PR
• Execute SCA (Snyk, Dependabot) for dependency vulnerabilities
• Scan container images (Trivy, Grype) before deployment
• Implement secrets scanning (Gitleaks, TruffleHog) in pre-commit hooks
• Use OIDC/Workload Identity instead of static credentials
• Sign artifacts with Sigstore/Cosign for supply chain integrity

3. Build Optimization

You will optimize pipeline performance:

• Implement intelligent caching (dependencies, build artifacts, Docker layers)
• Use matrix strategies for parallel test execution
• Configure incremental builds when possible
• Optimize Docker builds with multi-stage patterns
• Use build caching services (BuildKit, Kaniko)
• Profile and eliminate bottlenecks in build times

4. Deployment Automation

You will implement safe deployment strategies:

• Blue/green deployments for zero-downtime updates
• Canary deployments with progressive traffic shifting
• Rolling updates with proper health checks
• GitOps patterns with ArgoCD or Flux
• Automated rollback on failure detection
• Environment-specific configurations with proper isolation

5. Observability and Debugging

You will ensure pipeline visibility:

• Implement structured logging in all pipeline stages
• Track key metrics (build time, success rate, deployment frequency)
• Set up alerts for pipeline failures
• Create dashboards for build performance trends
• Implement proper error reporting and notifications
• Maintain audit trails for compliance

---

4. Top 7 Pipeline Patterns

Pattern 1: Secure Multi-Stage GitHub Actions Pipeline

# .github/workflows/ci-cd.yml
name: CI/CD Pipeline

on:
  pull_request:
    branches: [main, develop]
  push:
    branches: [main]

permissions:
  contents: read
  security-events: write
  id-token: write  # For OIDC

jobs:
  # Stage 1: Code Quality & Security
  code-quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for better analysis

      - name: Run Semgrep SAST
        uses: semgrep/semgrep-action@v1
        with:
          config: p/security-audit

      - name: SonarQube Scan
        uses: sonarsource/sonarqube-scan-action@master
        env:
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}

  # Stage 2: Dependency Scanning
  dependency-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Dependency Review
        uses: actions/dependency-review-action@v4
        with:
          fail-on-severity: high

      - name: Snyk Security Scan
        uses: snyk/actions/node@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

  # Stage 3: Build & Test
  build:
    runs-on: ubuntu-latest
    needs: [code-quality, dependency-check]
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run tests with coverage
        run: npm run test:coverage

      - name: Upload coverage
        uses: codecov/codecov-action@v3

      - name: Build application
        run: npm run build

      - name: Upload build artifacts
        uses: actions/upload-artifact@v4
        with:
          name: dist
          path: dist/
          retention-days: 7

  # Stage 4: Container Build & Scan
  container:
    runs-on: ubuntu-latest
    needs: build
    outputs:
      image-digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - name: Download build artifacts
        uses: actions/download-artifact@v4
        with:
          name: dist
          path: dist/

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to Container Registry (OIDC)
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push Docker image
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: |
            ghcr.io/${{ github.repository }}:${{ github.sha }}
            ghcr.io/${{ github.repository }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Scan image with Trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ghcr.io/${{ github.repository }}:${{ github.sha }}
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'

      - name: Upload Trivy results to GitHub Security
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

  # Stage 5: Sign Artifacts
  sign:
    runs-on: ubuntu-latest
    needs: container
    permissions:
      packages: write
      id-token: write
    steps:
      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Sign container image
        run: |
          cosign sign --yes \
            ghcr.io/${{ github.repository }}@${{ needs.container.outputs.image-digest }}

  # Stage 6: Deploy to Staging
  deploy-staging:
    runs-on: ubuntu-latest
    needs: sign
    if: github.ref == 'refs/heads/main'
    environment: staging
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to Kubernetes
        run: |
          kubectl set image deployment/myapp \
            myapp=ghcr.io/${{ github.repository }}:${{ github.sha }} \
            --namespace=staging

      - name: Wait for rollout
        run: |
          kubectl rollout status deployment/myapp \
            --namespace=staging \
            --timeout=5m

      - name: Run smoke tests
        run: npm run test:smoke -- --env=staging

  # Stage 7: Deploy to Production
  deploy-production:
    runs-on: ubuntu-latest
    needs: deploy-staging
    if: github.ref == 'refs/heads/main'
    environment: production
    steps:
      - uses: actions/checkout@v4

      - name: Deploy via ArgoCD
        run: |
          argocd app set myapp \
            --parameter image.tag=${{ github.sha }}
          argocd app sync myapp --prune
          argocd app wait myapp --health --timeout 600

Key Features:

• ✅ Security scans at multiple stages (SAST, SCA, container scanning)
• ✅ Proper dependency management with artifact passing
• ✅ OIDC authentication (no static secrets)
• ✅ Layer caching for Docker builds
• ✅ Artifact signing with Cosign
• ✅ Environment-specific deployments with approvals

📚 For more pipeline examples (GitLab CI, Jenkins, matrix builds, monorepo patterns):

• See [references/pipeline-examples.md](/home/user/ai-coding/new-skills/cicd-expert/references/pipeline-examples.md)

---

Pattern 2: Reusable Workflow for Microservices

# .github/workflows/reusable-service-build.yml
name: Reusable Service Build

on:
  workflow_call:
    inputs:
      service-name:
        required: true
        type: string
      node-version:
        required: false
        type: string
        default: '20'
      run-e2e-tests:
        required: false
        type: boolean
        default: false
    secrets:
      SONAR_TOKEN:
        required: true
      NPM_TOKEN:
        required: false

jobs:
  build-test-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ inputs.node-version }}
          cache: 'npm'
          cache-dependency-path: services/${{ inputs.service-name }}/package-lock.json

      - name: Install dependencies
        working-directory: services/${{ inputs.service-name }}
        run: npm ci

      - name: Run unit tests
        working-directory: services/${{ inputs.service-name }}
        run: npm run test:unit

      - name: Run integration tests
        if: inputs.run-e2e-tests
        working-directory: services/${{ inputs.service-name }}
        run: npm run test:integration

      - name: Build service
        working-directory: services/${{ inputs.service-name }}
        run: npm run build

      - name: SonarQube Analysis
        uses: sonarsource/sonarqube-scan-action@master
        env:
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
        with:
          projectBaseDir: services/${{ inputs.service-name }}

# Usage in caller workflow:
# jobs:
#   build-auth-service:
#     uses: ./.github/workflows/reusable-service-build.yml
#     with:
#       service-name: auth-service
#       run-e2e-tests: true
#     secrets:
#       SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

---

Pattern 3: Smart Caching Strategy

name: Optimized Build with Caching

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Cache npm dependencies
      - name: Cache npm modules
        uses: actions/cache@v3
        with:
          path: ~/.npm
          key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [martinholovsky](https://github.com/martinholovsky)
- **Source:** [martinholovsky/claude-skills-generator](https://github.com/martinholovsky/claude-skills-generator)
- **License:** Unlicense

Install and usage instructions live in the source repository linked above.