Runtime Config Flagd

Feature Flag Management (flagd)

How It Works

The OpenTelemetry Demo uses flagd as its feature flag provider. Flags are stored in a Kubernetes ConfigMap and loaded by the flagd service. Services check flags at runtime via the OpenFeature SDK to decide whether to inject failures.

Flag lifecycle:

1. Flag definitions live in a ConfigMap (flagd-config in otel-demo namespace)
2. flagd watches the ConfigMap and hot-reloads on changes
3. Services evaluate flags via gRPC and behave accordingly
4. To remediate an incident, set the flag's default variant to off

Available Scripts

All scripts are in .claude/skills/runtime-config-flagd/scripts/

list_scenarios.py - List Incident Scenarios (START HERE)

Shows all available incident scenarios with their current status.

python .claude/skills/runtime-config-flagd/scripts/list_scenarios.py

# Only show currently active scenarios
python .claude/skills/runtime-config-flagd/scripts/list_scenarios.py --active-only

# JSON output
python .claude/skills/runtime-config-flagd/scripts/list_scenarios.py --json

Output includes: flag name, current state (active/inactive), affected service, effect description, detection method, and remediation steps.

list_flags.py - List All Feature Flags

python .claude/skills/runtime-config-flagd/scripts/list_flags.py

# Show full variant details
python .claude/skills/runtime-config-flagd/scripts/list_flags.py --verbose

# Only incident-related flags
python .claude/skills/runtime-config-flagd/scripts/list_flags.py --incidents-only

get_flag.py - Inspect a Specific Flag

python .claude/skills/runtime-config-flagd/scripts/get_flag.py 

# Examples:
python .claude/skills/runtime-config-flagd/scripts/get_flag.py paymentServiceFailure
python .claude/skills/runtime-config-flagd/scripts/get_flag.py adServiceHighCpu --json

set_flag.py - Toggle a Flag (Remediation)

# ALWAYS dry-run first
python .claude/skills/runtime-config-flagd/scripts/set_flag.py   --dry-run

# Then apply
python .claude/skills/runtime-config-flagd/scripts/set_flag.py  

# Examples:
python .claude/skills/runtime-config-flagd/scripts/set_flag.py paymentServiceFailure off --dry-run
python .claude/skills/runtime-config-flagd/scripts/set_flag.py paymentServiceFailure off
python .claude/skills/runtime-config-flagd/scripts/set_flag.py adServiceHighCpu off
python .claude/skills/runtime-config-flagd/scripts/set_flag.py emailMemoryLeak off

Incident Scenarios Reference

| Scenario | Flag | Service | Effect | Remediation | |----------|------|---------|--------|-------------| | Payment Failure | paymentServiceFailure | payment (Node.js) | Configurable % of requests fail | Set to off | | Payment Unreachable | paymentServiceUnreachable | payment | Complete unavailability | Set to off | | High CPU | adServiceHighCpu | ad (Java) | CPU spike 80-100% | Set to off | | GC Pressure | adServiceManualGc | ad (Java) | Frequent full GC pauses | Set to off | | Ad Failure | adServiceFailure | ad (Java) | Ad service errors | Set to off | | Memory Leak | emailMemoryLeak | email (Ruby) | OOM after minutes | Set to off + restart pod | | Latency Spike | imageSlowLoad | image-provider | 5-10s delay | Set to off | | Kafka Lag | kafkaQueueProblems | checkout/accounting | Consumer lag | Set to off | | Cache Failure | recommendationServiceCacheFailure | recommendation | Cache miss storm | Set to off | | Catalog Failure | productCatalogFailure | product-catalog | Product query errors | Set to off | | Cart Failure | cartServiceFailure | cart (.NET) | Cart ops fail | Set to off | | Traffic Spike | loadgeneratorFloodHomepage | all services | Request flood | Set to off | | LLM Inaccuracy | llmInaccurateResponse | product-reviews | Wrong AI content | Set to off | | LLM Rate Limit | llmRateLimitError | product-reviews | 429 errors | Set to off |

Flag Variant Reference

Flags have different variant types:

Boolean flags (on/off):

• adServiceHighCpu, adServiceManualGc, adServiceFailure, paymentServiceUnreachable, cartServiceFailure
• recommendationServiceCacheFailure, productCatalogFailure
• llmInaccurateResponse, llmRateLimitError
• Variants: on (true), off (false)

Percentage flags (graduated failure rate):

• paymentServiceFailure: off (0), 10% (0.1), 25% (0.25), 50% (0.5), 75% (0.75), 90% (0.95), 100% (1)

Intensity flags (graduated effect):

• emailMemoryLeak: off (0), 1x (1), 10x (10), 100x (100), 1000x (1000), 10000x (10000)
• imageSlowLoad: off (0), 5sec (5000ms), 10sec (10000ms)

Numeric flags:

• kafkaQueueProblems: off (0), on (100)
• loadgeneratorFloodHomepage: off (0), on (100)

Version Compatibility

| Flag | otel-demo Version | Notes | |------|------------------|-------| | All flags except below | v1.11.1+ | Standard deployment | | emailMemoryLeak | > v1.11.1 | Not in v1.11.1 ConfigMap | | llmInaccurateResponse | > v1.11.1 | Requires product-reviews LLM | | llmRateLimitError | > v1.11.1 | Requires product-reviews LLM |

> Note: loadgeneratorFloodHomepage uses a lowercase 'g' in loadgenerator.

Remediation Workflow

1. Identify the scenario - Use list_scenarios.py --active-only to see active incidents
2. Confirm the flag - Use get_flag.py to verify current state
3. Dry-run - Use set_flag.py off --dry-run to preview change
4. Apply - Use set_flag.py off to disable the incident
5. Verify - Check metrics/logs to confirm the issue is resolving
6. Post-remediation - Some scenarios (memory leak) may also require a pod restart

Safety

• ALWAYS dry-run first before setting flags
• Setting a flag to off is always safe — it restores normal behavior
• Setting a flag to on or a non-zero variant injects failures — only do this intentionally
• Flag changes take effect within seconds (flagd hot-reload)
• For emailMemoryLeak, after setting to off you may also need to restart the email pod to reclaim leaked memory

Quick Commands

| Goal | Command | |------|---------| | See active incidents | list_scenarios.py --active-only | | Check a specific flag | get_flag.py paymentServiceFailure | | Disable payment failure | set_flag.py paymentServiceFailure off | | Enable payment failure at 50% | set_flag.py paymentServiceFailure 50% | | Disable all CPU spikes | set_flag.py adServiceHighCpu off | | See all flags verbose | list_flags.py --verbose |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: swapnildahiphale
• Source: swapnildahiphale/OpenSRE
• License: Apache-2.0
• Homepage: https://opensre.in

Install and usage instructions live in the source repository linked above.