Galileo Lemonade Instrumentation Setup

Galileo Lemonade Instrumentation Setup

Prerequisites

| Tool or access | Purpose | Verify | |---|---|---| | Bash and Python 3 | Run bundled setup and validation helpers | bash --version && python3 --version | | Required product/platform access | Inspect or configure the selected target | Complete the documented preflight | | Credential files for live modes | Keep secrets out of chat | Verify paths only |

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Adding Galileo OTLP fan-out to a Lemonade collector, capturing agent/workflow/tool traces around Lemonade,

avoiding duplicate LLM records, or validating privacy-safe Galileo ingestion from an AMD Ryzen AI host.

• Preview and review the galileo lemonade instrumentation setup workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/galileo-lemonade-instrumentation-setup/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/galileo-lemonade-instrumentation-setup/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

| Issue | Cause | Resolution | |---|---|---| | Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying | | Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again | | Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff | | Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |

Purpose

Use this skill after $lemonade-splunk-otel has established a healthy, privacy-safe Lemonade-to-Splunk trace path. It adds Galileo in one of two explicit modes and proves delivery by backend readback.

The tested baseline is Lemonade v10.10 with Splunk OTel Collector v0.156; Galileo SaaS and Enterprise still require an exact tenant intake endpoint. Run the tools from an isolated Python environment containing requirements-dev.txt (PyYAML 6.x). Rendering is safe offline; production validation also requires the exact installed collector binary.

This skill does not treat Lemonade's native inference span as a complete AI agent trace. Lemonade v10.10 creates one root SERVER span per request, does not extract traceparent, and has no workflow/tool hierarchy.

Required Intake

Ask for and record the exact Galileo instance console URL. A copied console link may include a navigation route after the host. Validate that exact link with --galileo-console-url, but pass only its reported HTTPS origin to $galileo-platform-setup. For example, https://console.example.invalid/tenant-navigation normalizes to https://console.example.invalid/ for endpoint derivation. Treat the path as navigation context; never infer a project or Log stream from it. The demo-v2 deployment used by the production example starts at https://console.demo-v2.galileocloud.io/; still require and validate the operator's full navigation URL instead of silently assuming that instance.

Resolve the API base, exact OTLP traces endpoint, project, and Log stream independently. The renderer reports an endpoint candidate but keeps the runtime value in GALILEO_OTLP_TRACES_ENDPOINT until tenant validation.

Choose One Galileo Source

| Requirement | Mode | Galileo receives | Privacy consequence | |---|---|---|---| | No application change; model/latency/token metadata | server-fanout | Lemonade's native LLM root spans | With Lemonade hide flags on, input/output/thinking stay [REDACTED]; content evaluators are limited. | | Agent, workflow, tool, session, async, or selective evaluation context | client-fanout | Caller-side OpenInference/OTel spans, Galileo-only by default | Recommended for actual agent observability. Native Lemonade remains redacted and Splunk-only. | | Remove Galileo routing without disturbing Splunk | splunk-only | Nothing | Rollback/render cleanup mode. |

Do not send both native and caller-instrumented LLM spans to the same Galileo Log stream. They have unrelated trace IDs and appear as duplicate observations. If temporary comparison is required, use separate Log streams and label it.

Direct Lemonade-to-Galileo is technically possible but is not the default: the server has one OTLP destination, so it displaces Splunk and exposes the Galileo key to the Lemonade service.

Safety Contract

• Validate the exact Galileo console URL, normalize a copied navigation link

to its origin, and do not assume public Galileo Cloud.

• Use $galileo-platform-setup for tenant readiness and project/Log stream

lifecycle. Prefer immutable project and Log stream IDs after discovery.

• Require the exact tenant-supported traces_endpoint. Current Galileo docs

show both /otel/v1/traces for raw OTLP POST and /otel/traces for several exporter/SDK integrations. Do not derive one from the client type or let the collector append a path.

• Stock Collector v0.156 follows HTTP redirects and can copy custom headers to

the redirect target. Pin the endpoint to a separately discovered GALILEO_EXPECTED_ORIGIN, then route only otlp_http/galileo_lemonade through the literal Collector v0.156 proxy_url field. The proxy must be a dedicated loopback tinyproxy with FilterDefaultDeny Yes; its only filter rule is the anchored, regex-escaped exact host derived from that pinned origin. Production revalidates protected binary/config/filter path identity, inode, metadata, and SHA-256, then runs bounded credential-free CONNECT probes that require HTTP 403 for an unlisted host and 2xx for the exact Galileo host. All Splunk exporters remain direct. Ambient proxy variables are stripped and are never part of this contract. Install evidence as root-owned, collector-group-readable mode 0440; a root 0400 file cannot be read by the non-root collector wrapper.

• Use the current collector component type otlp_http, not legacy

otlphttp. This skill names its instance otlp_http/galileo_lemonade so it cannot overwrite another application's Galileo exporter.

• Keep Galileo credentials in a dedicated service-user-owned 0600 file

(root-owned only when the collector runs as root). Generated YAML contains only ${env:...} placeholders.

• Use scripts/galileo_bootstrap_transaction.py when a broad bootstrap key

must create or adopt the exact target and mint the project-scoped runtime key. The bootstrap secret is accepted only from a current-user-owned, single-link protected file; it is never accepted through argv or the environment. Keep the private journal and one-time runtime-key output until the transaction is finalized or rolled back.

• Bootstrap stops at RUNTIME_KEY_CREATED. Never revoke the old key in that

invocation. Record fresh, exact cutover evidence and run the separate finalize command with a distinct reviewed unscoped revoker credential only after host cutover, Galileo API trace/hierarchy and privacy readback, and unchanged Splunk backend readback all pass. Console UI review is not inferred from API evidence and is not a revocation gate.

• Use GALILEO_API_KEY_FILE with the packaged collector runtime wrapper. Do

not source a plaintext key into an interactive shell or store it in the non-secret collector environment file.

• Pin the collector the wrapper may exec with GALILEO_COLLECTOR_BINARY and

GALILEO_COLLECTOR_BINARY_SHA256. Exec mode requires both and fails closed without them, before the API key is read. Exec mode also requires Linux and refuses to run elsewhere, because the binary and every ancestor directory must be proven root-owned, link-free, and not group/other-writable. Recompute the digest after every collector package change.

• Render one complete config from the live base, review its diff, validate with

the exact installed collector binary, back up, then apply transactionally.

• Keep both Lemonade and client OTLP receivers loopback-bound.
• Treat service.name filtering as classification, not authentication. Use

server-fanout only within a loopback/single-host trust boundary and accept its shared-receiver replay risk explicitly during production validation.

• Use the persistent Galileo queue for production. The memory queue is an

accepted-loss development option and cannot pass --production validation.

• Bind every persistent queue to the SHA-256 of its validated endpoint and

selector pair. The queue directory's final component must be that fingerprint; never reuse, rename, or copy it to a different destination.

• Delete every galileo.* resource, span, and event attribute immediately

before the Galileo exporter so in-band project, Log stream, experiment, or dataset fields cannot override the fixed exporter headers.

• Never disable content hiding without explicit approval of every backend that

will receive the affected pipeline.

Workflow

1. Read [reference.md](reference.md), then load the architecture, application,

or validation reference required by the chosen mode.

1. Run $lemonade-splunk-otel discovery and confirm the existing Splunk path,

privacy flags, native trace pipeline name, collector config, and receiver.

1. Run $galileo-platform-setup readiness for the user-confirmed instance.

Put the existing bootstrap key in a protected file without printing it and identify its exact API-key ID. For read-only inventory, resolve immutable project and Log stream IDs without creating objects:

``bash python3 skills/galileo-lemonade-instrumentation-setup/scripts/galileo_target_discovery.py \ --api-base "$GALILEO_API_BASE" \ --api-key-file "$GALILEO_API_KEY_FILE" \ --api-key-header Splunk-AO-API-Key ``

Filter with exact --project-name or --project-id when the tenant has many projects. Current v2 project/read APIs document Splunk-AO-API-Key; OTLP ingest separately uses Galileo-API-Key. Confirm the selected Log stream before rendering.

When the target or project-scoped runtime key must be created, use the phased bootstrap transaction in [references/runtime-credentials.md](references/runtime-credentials.md). Existing targets require explicit adoption, preferably by exact IDs. The default candidate role is annotator; it is accepted only if a live API probe proves exact project-only visibility and log_data. If that probe fails, roll back the exact owned key and start a new transaction before trying editor; do not claim either role is least privilege without the live permission proof.

1. State the selected Galileo source and content policy before rendering.
2. With the confirmed endpoint, expected origin, and exactly one selector pair

set in the protected runtime environment, calculate the non-secret destination fingerprint. This command emits only the lowercase digest and does not require or print the API key:

``bash python3 skills/galileo-lemonade-instrumentation-setup/scripts/collector_runtime_wrapper.py \ --print-destination-fingerprint ``

Record it as GALILEO_DESTINATION_FINGERPRINT, and set GALILEO_QUEUE_STORAGE_DIRECTORY to a new private directory ending with that exact digest.

Pin the collector the wrapper may exec. Record the reviewed binary as GALILEO_COLLECTOR_BINARY and its digest as GALILEO_COLLECTOR_BINARY_SHA256, read from the installed file itself, not from a vendor download page:

``bash sha256sum /usr/bin/otelcol ``

Render from the existing full collector config:

``bash bash skills/galileo-lemonade-instrumentation-setup/scripts/setup.sh \ --galileo-console-url "$GALILEO_CONSOLE_URL" \ --base /etc/otel/collector/agent_config.yaml \ --output /tmp/lemonade-galileo-agent_config.yaml \ --mode client-fanout \ --routing ids \ --galileo-proxy-url http://127.0.0.1:18888 \ --queue-policy persistent \ --production \ --destination-fingerprint "$GALILEO_DESTINATION_FINGERPRINT" \ --queue-storage-directory "$GALILEO_QUEUE_STORAGE_DIRECTORY" ``

1. Review the diff, then apply four independently journaled layers in dependency

order: create the destination-fingerprinted directory with transactional_queue_directory.py; install and probe the dedicated tinyproxy package/config/filter/unit with transactional_proxy_bundle.py; render protected proxy identity evidence from those installed assets; and install the routing environment, evidence, wrapper, key, and drop-in with transactional_runtime_bundle.py. Follow [references/queue-directory-transaction.md](references/queue-directory-transaction.md), [references/proxy-bundle-transaction.md](references/proxy-bundle-transaction.md), [references/runtime-bundle-transaction.md](references/runtime-bundle-transaction.md), after reading the underlying credential contract in [references/runtime-credentials.md](references/runtime-credentials.md). Do not apply the collector YAML yet. Start the proxy and run the wrapper's --check; it must prove the protected assets plus credential-free live allow/deny probes. Then validate the staged YAML statically and with the installed collector:

Use $lemonade-splunk-otel's value-free config_change_summary.py first; PyYAML normalizes formatting and comments, so retain the exact source backup.

The production validator inspects service-owned queue files and collector-group-readable proxy evidence. Run the command under the exact Collector UID, primary GID, and supplementary groups, while inheriting the protected systemd environment without copying secret values into argv. Do not run it as root: root's group set is not proof that the Collector can read or safely own those assets.

``bash # Execute as the discovered Collector service identity, not as root. bash skills/galileo-lemonade-instrumentation-setup/scripts/validate.sh \ --collector-config /tmp/lemonade-galileo-agent_config.yaml \ --mode client-fanout \ --queue-policy persistent \ --production \ --galileo-proxy-url http://127.0.0.1:18888 \ --destination-fingerprint "$GALILEO_DESTINATION_FINGERPRINT" \ --queue-storage-directory "$GALILEO_QUEUE_STORAGE_DIRECTORY" \ --collector-binary /usr/bin/otelcol ``

1. Back up the live collector config and service state, preserving the exact

original command/arguments. After the three prerequisite transactions and service-identity validation pass, use the baseline skill's SHA-gated transactional apply helper for the validated YAML and restart, with /etc/splunk-otel-collector/lemonade-agent-config.yaml as the live config path pinned by the runtime manifest. The wrapper validates endpoint/origin, selectors, destination fingerprint, proxy assets/live probes, and queue before loading the protected key only in the collector child. Restore the collector YAML transaction immediately if Splunk regresses, then restore the exact Collector YAML manifest, runtime manifest, proxy manifest, and queue manifest

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: chambear2809
• Source: chambear2809/splunk-cisco-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.