Bottlerocket Diagnostics

Bottlerocket Diagnostics

When to use

Any Bottlerocket investigation — AMI launch failures, OS update issues, ECS agent problems, kubelet errors, admin/control container access, networking configuration, storage issues, settings API errors, bootstrap failures, or security hardening.

Investigation workflow

Step 1 — Collect and triage

aws ec2 describe-instances --filters "Name=tag:OperatingSystem,Values=Bottlerocket" --query 'Reservations[*].Instances[*].{Id:InstanceId,State:State.Name,ImageId:ImageId,LaunchTime:LaunchTime}'
aws ssm describe-instance-information --filters "Key=PlatformType,Values=Linux" --query 'InstanceInformationList[*].{Id:InstanceId,PingStatus:PingStatus,Platform:PlatformName,Version:PlatformVersion}'
aws ec2 describe-images --owners amazon --filters "Name=name,Values=bottlerocket-aws-k8s-*" --query 'Images | sort_by(@, &CreationDate) | [-5:].{Name:Name,ImageId:ImageId,Created:CreationDate}'

Step 2 — Domain deep dive

aws ssm start-session --target  --document-name AWS-StartInteractiveCommand --parameters command="apiclient get settings"
aws ecs describe-container-instances --cluster  --container-instances  --query 'containerInstances[*].{Status:status,Agent:agentConnected,Version:versionInfo}'
aws eks describe-nodegroup --cluster-name  --nodegroup-name  --query 'nodegroup.{Status:status,AmiType:amiType,Version:version}'

Step 3 — Detailed investigation

aws ec2 get-console-output --instance-id  --latest
aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name StatusCheckFailed --dimensions Name=InstanceId,Value= --start-time  --end-time  --period 300 --statistics Maximum
aws ssm start-session --target  --document-name AWS-StartInteractiveCommand --parameters command="logdog"

Read references/guardrails.md before concluding on any Bottlerocket issue.

Tool quick reference

| Tool / API | When to use | |------------|-------------| | ec2 describe-instances | List Bottlerocket instances and status | | ec2 describe-images | Find available Bottlerocket AMIs | | ec2 get-console-output | Retrieve boot logs for diagnosis | | ssm start-session | Access admin or control container | | ecs describe-container-instances | Check ECS agent connectivity | | eks describe-nodegroup | Check EKS node group health | | apiclient get settings | Read Bottlerocket settings via API |

Gotchas: Bottlerocket

• Bottlerocket has NO package manager and NO shell by default. The admin container must be enabled for SSH-like access. The control container provides SSM access.
• Bottlerocket uses an API-driven settings model (apiclient). Configuration files are NOT directly editable — all changes go through the settings API.
• Updates use a dual-partition (A/B) scheme. The inactive partition is updated, then the system reboots into it. Rollback swaps back to the previous partition.
• ECS and EKS variants use DIFFERENT AMIs. Using the wrong variant (e.g., k8s AMI for ECS) causes agent failures.
• The admin container is disabled by default for security. Enabling it requires user data or SSM. It provides sheltie to access the host filesystem.
• Bottlerocket logs are collected via logdog command in the control container, NOT via traditional /var/log paths.
• SELinux is enforced by default. Custom configurations that bypass SELinux will fail silently.

Anti-hallucination rules

1. Always cite specific instance IDs, AMI IDs, or API responses as evidence.
2. ECS variant and EKS variant are completely different AMIs. Never suggest using one for the other.
3. Bottlerocket has NO yum, apt, or any package manager. Never suggest installing packages.
4. Settings are API-driven only. Never suggest editing config files directly on the host.
5. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.

12 runbooks

| Category | IDs | Covers | |----------|-----|--------| | A — AMI & Updates | A1–A2 | AMI launch failures, update failures | | B — Container Agents | B1–B2 | ECS agent issues, kubelet issues | | C — Access | C1–C2 | Admin container access, control container issues | | D — Infrastructure | D1–D2 | Networking configuration, storage issues | | E — Configuration | E1–E2 | Settings API errors, bootstrap failures | | F — Security | F1 | Security hardening | | Z — Catch-All | Z1 | General troubleshooting |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: aws-samples
• Source: aws-samples/sample-ai-agent-skills
• License: MIT-0

Install and usage instructions live in the source repository linked above.