Batch Diagnostics

AWS Batch Diagnostics

When to use

Any AWS Batch investigation where the console alone is insufficient — job failures, compute environment issues, scheduling problems, container configuration, networking, array/multi-node jobs, performance optimization, or security troubleshooting.

Investigation workflow

Step 1 — Collect and triage

aws batch describe-compute-environments --compute-environments 
aws batch describe-job-queues --job-queues 
aws batch describe-jobs --jobs 
aws batch list-jobs --job-queue  --job-status FAILED --max-results 10
aws batch list-jobs --job-queue  --job-status RUNNABLE --max-results 10

Step 2 — Domain deep dive

aws batch describe-job-definitions --job-definition-name  --status ACTIVE
aws batch describe-jobs --jobs  --query 'jobs[*].{Status:status,Reason:statusReason,Container:container.{ExitCode:exitCode,Reason:reason,LogStream:logStreamName}}'
aws logs get-log-events --log-group-name /aws/batch/job --log-stream-name 
aws ec2 describe-instances --filters Name=tag:aws:batch:compute-environment,Values=

Step 3 — Detailed investigation

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=batch.amazonaws.com --max-results 20
aws ecs describe-clusters --clusters 
aws cloudwatch get-metric-statistics --namespace AWS/Batch --metric-name CPUUtilization --dimensions Name=JobQueue,Value= --start-time  --end-time  --period 300 --statistics Average

Read references/batch-guardrails.md before concluding on any AWS Batch issue.

Tool quick reference

| Tool / API | When to use | |------------|-------------| | describe-jobs | Check job status, exit code, reason, log stream | | list-jobs | List jobs by status in a queue | | describe-compute-environments | Check CE config, state, instance types | | describe-job-queues | Check queue priority and CE associations | | describe-job-definitions | Check container config, resources, IAM | | CloudWatch Logs | Check container stdout/stderr logs | | ECS | Check underlying ECS cluster and tasks | | EC2 | Check instances in managed CE |

Gotchas: AWS Batch

• RUNNABLE status means waiting for resources, not running. Jobs in RUNNABLE are waiting for compute capacity. This can be caused by: CE not scaling, instance types unavailable, VPC/subnet issues, or resource requirements exceeding CE limits. RUNNABLE is the most common stuck state.
• EC2 and Fargate compute environments have different capabilities. EC2 supports GPUs, custom AMIs, placement groups, and more instance types. Fargate is serverless but has vCPU/memory limits and no GPU support. Fargate Spot is available for cost savings.
• Job exit codes indicate failure type. Exit code 0 = success. Non-zero = application failure. Exit code 137 = OOM killed (container exceeded memory). Exit code 1 = general application error. Check container logs for details.
• Spot interruptions can terminate running jobs. Spot instances can be reclaimed with 2-minute warning. Batch retries Spot-interrupted jobs automatically (if retry strategy configured). Use a mix of Spot and On-Demand for critical workloads.
• Array jobs share a single job definition. Each child job gets an AWSBATCHJOBARRAYINDEX environment variable. Array size can be 2-10,000. Failed children don't affect other children. Dependencies can be set between array jobs.
• Multi-node parallel jobs require EC2 compute. They use ECS task placement for node coordination. The main node (index 0) coordinates work. All nodes must be able to communicate. Fargate does not support multi-node parallel.
• Job queue priority determines scheduling order. Higher priority queues are evaluated first. Jobs within a queue are FIFO. Fair-share scheduling can override priority-based scheduling. A queue can be associated with multiple CEs.
• Container resource requests must match CE capabilities. vCPU and memory requests must be available in the CE instance types. Requesting more resources than any instance type can provide causes RUNNABLE stuck. GPU requests require GPU instance types.

Anti-hallucination rules

1. Always cite specific job IDs, exit codes, or API responses as evidence.
2. RUNNABLE means waiting for resources, NOT running. Never confuse the two.
3. Exit code 137 means OOM killed. Never ignore memory-related exit codes.
4. Fargate does not support GPUs or multi-node parallel. Never suggest these.
5. Spot interruptions are expected. Never treat them as unexpected failures.
6. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.

24 runbooks

| Category | IDs | Covers | |----------|-----|--------| | A — Jobs | A1-A3 | Job failures, stuck RUNNABLE, timeout | | B — Compute | B1-B3 | Compute environment failures, EC2 vs Fargate, spot interruptions | | C — Queue | C1-C2 | Job queue issues, scheduling | | D — Definition | D1-D2 | Job definition errors, container config | | E — Networking | E1-E2 | VPC config, ECR access | | F — Array/Multi | F1-F2 | Array jobs, multi-node parallel | | G — Performance | G1-G2 | Throughput, resource optimization | | H — Security | H1-H2 | IAM roles, secrets | | Z — Catch-All | Z1 | General troubleshooting |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: aws-samples
• Source: aws-samples/sample-ai-agent-skills
• License: MIT-0

Install and usage instructions live in the source repository linked above.