Aws Log Analyzer

AWS Log Analyzer

Overview

Troubleshoot and analyze logs from AWS CloudWatch Logs - AWS's centralized logging service for applications and infrastructure.

Prerequisites:

• aws CLI is installed and configured
• User is already authenticated (via IAM credentials, SSO, or instance profile)
• Appropriate IAM permissions for CloudWatch Logs read operations

Installation: Use the centralized tool installation scripts:

# Check and install AWS CLI (required)
../../../tools/aws-cli/install.sh

# Check and install jq (optional, recommended)
../../../tools/jq/install.sh

Core Concepts

• Log Group: Container for log streams (typically one per application/service)
• Log Stream: Sequence of log events from a single source (e.g., instance, container)
• Log Event: Individual log entry with timestamp and message
• CloudWatch Logs Insights: SQL-like query language for advanced log analysis

Analysis Philosophy

Always follow this pattern:

1. Start broad → identify the problem scope
2. Narrow down → focus on specific errors or patterns
3. Filter noise → exclude known non-critical errors
4. Analyze distribution → understand when errors occur

Use CloudWatch Logs Insights for all error analysis - it supports case-insensitive regex, which is essential because logs may contain "error", "Error", or "ERROR" in different formats.

Output Format

All scripts output JSON by default to make results easy to parse programmatically by AI assistants and automation tools.

⚠️ RECOMMENDATION: Use full JSON output for typical error analysis

For most use cases, direct JSON output is more efficient than state management:

• ✅ Single round-trip - get all data in one call
• ✅ No state lookup complexity - data is immediately available
• ✅ Reliable - no session ID or file path issues
• ✅ For typical analyses (even 10K+ errors), JSON output is manageable (~30KB)

Only use --save-state (via state management scripts) if:

• You're analyzing 100K+ log entries
• The JSON output exceeds 100KB
• You need to reference the same data across multiple analysis steps over time

Recommended approach - get all data in one call:

# Run analysis and capture full JSON output
OUTPUT=$(./scripts/analyze_errors.sh  24)

# Parse specific fields as needed
echo "$OUTPUT" | jq '.total_errors'
echo "$OUTPUT" | jq '.by_severity'
echo "$OUTPUT" | jq '.top_errors[:5]'

JSON Output (Default)

# Default: JSON output to stdout, progress to stderr
./scripts/analyze_errors.sh /aws/app/myapp 24

# Output:
{
  "log_group": "/aws/app/myapp",
  "hours_analyzed": 24,
  "total_errors": "1247",
  "by_severity": {
    "critical": 15,
    "error": 1200,
    "warning": 25,
    "failed": 7
  },
  "top_errors": [
    {
      "message": "Connection timeout to database",
      "count": 342,
      "percentage": 27.43,
      "pattern": "Connection timeout to database"
    },
    ...
  ],
  "critical_errors": [...],
  "top_errors_by_pattern": [
    {
      "pattern": "Error at ",
      "total_count": 450,
      "occurrences": 12,
      "examples": [
        {"message": "Error at 2026-02-06 15:30:45", "count": 120},
        {"message": "Error at 2026-02-06 16:45:12", "count": 95}
      ]
    },
    ...
  ],
  "hourly_distribution": [...],
  "comparison": null  // or populated if --compare-previous is used
}

With additional flags:

# Exclude noise patterns and compare with previous period
./scripts/analyze_errors.sh /aws/app/myapp 24 --exclude-noise --compare-previous

# Output includes comparison data:
{
  ...
  "comparison": {
    "current_period": {"total_errors": 1247, "hours": 24},
    "previous_period": {"total_errors": 1050, "hours": 24},
    "change": "+18.76%",
    "trend": "increasing"
  }
}

Benefits:

• Structured data - Easy to parse and extract specific fields
• Clean separation - Progress messages go to stderr, results to stdout
• Consistent format - All scripts use the same JSON structure pattern
• AI-friendly - Models can easily process and reason about JSON

Human-Readable Output

Add --human flag for human-readable table/text format:

./scripts/analyze_errors.sh /aws/app/myapp 24 --human

# Output:
=== Error Analysis Results ===
Log Group: /aws/app/myapp
Total Errors: 1247

Top Errors by Frequency:
  342x: Connection timeout to database
  125x: Authentication failed...
  ...

Parsing JSON Output

In the model context:

# Extract specific field
./scripts/analyze_errors.sh /aws/app/myapp 24 | jq '.total_errors'

# Count distinct error types
./scripts/analyze_errors.sh /aws/app/myapp 24 | jq '.top_errors | length'

# Get top 3 errors
./scripts/analyze_errors.sh /aws/app/myapp 24 | jq '.top_errors[:3]'

---

State Management

⚠️ IMPORTANT: State management is for advanced use cases only

The state management system (~/.aws-log-analyzer/state/) is available for very large datasets or multi-step workflows, but is NOT recommended for typical error analysis.

Shared Library: This skill uses claudio-plugin/tools/memory/scripts/state.sh - a shared state management library used across multiple skills.

Why direct JSON output is better:

• ✅ Simpler - no session IDs or file paths to manage
• ✅ Faster - single round-trip instead of save → view → parse
• ✅ More reliable - no file system dependencies
• ✅ Efficient even for 10K+ errors (~30KB JSON)

Note: analyze_errors.sh no longer supports the --save-state flag. Use direct JSON output instead (see examples above).

Manual State Management (Advanced)

If you need state management for very large datasets (100K+ entries), you can manually save/load data:

# Capture output and save manually if needed
OUTPUT=$(./scripts/analyze_errors.sh  24)
echo "$OUTPUT" > /tmp/analysis_result.json

# Later, load and parse
jq '.top_errors[:10]' /tmp/analysis_result.json

View saved state:

# List all saved states
./scripts/view_state.sh

# View specific state by ID
./scripts/view_state.sh analyze_errors_1707224567

# View latest state for an operation
./scripts/view_state.sh analyze_errors

When to Use State

Use --save-state when:

• Working with the model and want to minimize token usage
• Results are large (thousands of log entries, many error types)
• Building multi-step workflows where later steps reference earlier results

Don't use --save-state when:

• Running scripts manually and want immediate full output
• Results are small and fit easily in context
• Doing one-off investigations

Querying Saved State

Once data is saved, you can extract specific information using jq:

# Get the total error count
./scripts/view_state.sh analyze_errors | jq '.total_errors'

# Get top 5 errors with their counts
./scripts/view_state.sh analyze_errors | jq '.top_errors[0:5][] | {message: .message, count: .count}'

# Get only critical errors (count > 10)
./scripts/view_state.sh analyze_errors | jq '.critical_errors[] | select(.count > 10)'

# Get severity breakdown
./scripts/view_state.sh analyze_errors | jq '.by_severity'

# Get errors from a specific time bucket
./scripts/view_state.sh analyze_errors | jq '.hourly_distribution[] | select(.time_bucket | contains("2026-02-06T15"))'

# Extract error patterns (grouped by similarity)
./scripts/view_state.sh analyze_errors | jq '.top_errors_by_pattern[0:5]'

# Get all errors matching a specific pattern
./scripts/view_state.sh analyze_errors | jq '.top_errors[] | select(.pattern | contains("Connection"))'

# Get percentage of errors that are critical
./scripts/view_state.sh analyze_errors | jq '(.by_severity.critical / (.total_errors | tonumber) * 100)'

# Compare current vs previous period (if --compare-previous was used)
./scripts/view_state.sh analyze_errors | jq '.comparison'

# Get examples of a specific error pattern
./scripts/view_state.sh analyze_errors | jq '.top_errors_by_pattern[0].examples'

Example workflow using saved state:

# Step 1: Analyze errors with state saving
./scripts/analyze_errors.sh /aws/app/myapp 24 --save-state --exclude-noise

# Output:
# {
#   "operation": "analyze_errors",
#   "state_saved": true,
#   "state_id": "analyze_errors_1738858234",
#   "summary": {
#     "log_group": "/aws/app/myapp",
#     "total_errors": 1247,
#     "top_error_patterns": [...]
#   }
# }

# Step 2: Query specific details without re-running analysis
./scripts/view_state.sh analyze_errors | jq '.by_severity'
# Output: {"critical": 15, "error": 1200, "warning": 25, "failed": 7}

# Step 3: Get top 3 error patterns
./scripts/view_state.sh analyze_errors | jq '.top_errors_by_pattern[0:3]'

# Step 4: Find all errors with high frequency (> 50 occurrences)
./scripts/view_state.sh analyze_errors | jq '.top_errors[] | select(.count > 50)'

---

Available Scripts

All operations are performed through the following scripts:

State Management Scripts

• view_state.sh - View saved script outputs (for advanced workflows only)
• Note: analyze_errors.sh no longer supports --save-state flag
• State management is available through manual save/load if needed for very large datasets

Discovery Scripts

• list_log_groups.sh - List available log groups
• list_log_streams.sh - List log streams within a group

Analysis Scripts

• analyze_errors.sh - Complete error analysis (recommended for most cases)
• Flags: --human, --exclude-noise, --compare-previous
• Features: Severity classification, pattern grouping, trend analysis
• Output: Full JSON by default (efficient for typical datasets)
• find_recent_errors.sh - Quick search for recent errors
• run_insights_query.sh - Execute custom CloudWatch Logs Insights queries
• trace_request.sh - Trace a request ID across multiple log groups

Monitoring Scripts

• tail_logs.sh - Monitor logs in real-time

Template Queries

Pre-built CloudWatch Logs Insights queries are available in scripts/insights_queries.json:

Error Analysis:

• error_analysis.total_count - Count total errors
• error_analysis.by_message - Group errors by message
• error_analysis.unique_errors - Find unique errors (excludes noise)
• error_analysis.hourly_distribution - Hourly error distribution
• error_analysis.recent_errors - Last 100 errors

Performance Analysis:

• performance_analysis.slow_requests - Requests slower than 1s
• performance_analysis.latency_percentiles - P50, P90, P99 latencies
• performance_analysis.requests_per_minute - Request rate

Request Tracing:

• request_tracing.by_request_id - Trace by request ID
• request_tracing.by_user - Trace by user ID

Application Monitoring:

• application_monitoring.status_codes - HTTP status code distribution
• application_monitoring.error_rate - Error rate percentage
• application_monitoring.top_endpoints - Most accessed endpoints

Common Workflows

Workflow 1: Analyze Errors in a Log Group

User Request: "Analyze errors for in the last 24 hours"

Execution Sequence:

# Step 1: Run complete error analysis
./scripts/analyze_errors.sh  24

Output Provides:

1. Total error count
2. Top error messages by frequency
3. Critical/unique errors (excludes noise)
4. Hourly error distribution

Recommended approach - capture output and parse as needed:

# Step 1: Run analysis and capture full JSON output
OUTPUT=$(./scripts/analyze_errors.sh  24)

# Step 2: Extract specific fields
echo "$OUTPUT" | jq '.total_errors'
# Output: "1247"

echo "$OUTPUT" | jq '.by_severity'
# Output: {"critical": 15, "error": 1200, "warning": 25, "failed": 7}

echo "$OUTPUT" | jq '.top_errors[:3]'
# Output: Array of top 3 errors with counts and percentages

# Step 3: Get more details on specific errors if needed
./scripts/find_recent_errors.sh  1 50

Why this is efficient:

• Single execution of analyze_errors.sh gets all data
• No round-trips to view state
• No session ID management
• For typical datasets (even 10K errors), JSON is ~30KB - completely manageable
• Parse different fields from the same output as needed

---

Workflow 2: Investigate Errors (Unknown Log Group)

User Request: "Check for errors in my application"

Execution Sequence:

# Step 1: Find the log group
./scripts/list_log_groups.sh /aws/application

# Step 2: Analyze errors in the identified log group
./scripts/analyze_errors.sh  24

---

Workflow 3: Trace a Request Across Services

User Request: "Trace request ID abc-123 through all services"

Execution Sequence:

# Single command to search all log groups with a common prefix
./scripts/trace_request.sh abc-123 /aws/myapp 24

Output: Shows all log entries containing the request ID, sorted by timestamp, across all log groups.

---

Workflow 4: Monitor for Specific Errors in Real-Time

User Request: "Watch for OutOfMemory errors in real-time"

Execution Sequence:

# Tail logs with filter pattern
./scripts/tail_logs.sh  "OutOfMemoryError" 1h

Time formats: 1h, 30m, 2d, 5s

---

Workflow 5: Custom Error Analysis

User Request: "Find all authentication failures in the last 6 hours"

Execution Sequence:

# Step 1: Run custom Insights query
./scripts/run_insights_query.sh  6 \
  'fields @timestamp, @message | filter @message like /(?i)(auth|authentication)/ and @message like /(?i)(fail|denied)/ | sort @timestamp desc | limit 100'

Alternative using template query:

# Step 1: Load query from template (if you have one defined)
QUERY=$(jq -r '.custom_queries.auth_failures' scripts/insights_queries.json)

# Step 2: Run the query
./scripts/run_insights_query.sh  6 "$QUERY"

---

Workflow 6: Analyze Performance Issues

User Request: "Find slow database queries in the last 24 hours"

Execution Sequence:

# Step 1: Use performance template query
QUERY=$(jq -r '.performance_analysis.slow_requests' scripts/insights_queries.json)

# Step 2: Run the query
./scripts/run_insights_query.sh  24 "$QUERY"

For RDS slow query logs:

# Custom query for RDS slow query format
./scripts/run_insights_query.sh /aws/rds/instance/mydb/slowquery 24 \
  'fields @timestamp, query_time, lock_time, rows_examined, @message | parse @message /Query_time: (?[0-9.]+)\s+Lock_time: (?[0-9.]+).*\n(?.*)/ | filter qt > 1.0 | sort qt desc | limit 20'

---

Workflow 7: Investigate Recent Activity

User Request: "What's happening in my application right now?"

Execution Sequence:

# Step 1: List recent log streams to see activity
./scripts/list_log_streams.sh  10

# Step 2: Tail recent logs
./scripts/tail_logs.sh  "" 10m

# Step 3: If errors are seen, analyze them
./scripts/analyze_errors.sh  1

---

Workflow 8: Compare Error Rates

User Request: "Has the error rate increased in the last hour?"

Execution Sequence:

# Step 1: Get errors from last hour
./scripts/analyze_errors.sh  1

# Step 2: Get errors from previous hour for comparison
./scripts/run_insights_query.sh  2 \
  'fields @timestamp | filter @message like /(?i)(error|fail|exception|critical)/ | stats count() as error_count by bin(1h)'

---

Performance Optimization

When combining this skill with other skills (especially gitlab-job-analyzer):

See the complete optimization guide in the main CLAUDE.md documentation under "Performance Optimization for Cross-Skill Analysis".

Key optimizations:

1. Parallel execution - Run GitLab + AWS analysis simultaneously in one message
2. Parse JSON once - Capture output, parse multiple times with jq (don't re-run scripts)
3. Smart targeting - Analyze only log groups for failing runners/components identified in GitLab analysis
4. Direct JSON output - For typical analyses (10K+ errors), direct JSON is more efficient than state management

**Example - Optimized cross-skill an

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: aipcc-cicd
• Source: aipcc-cicd/claudio-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.