Pm Analytics

Product Analytics Skill

How this skill behaves (read first)

This is a generative process skill (it plans or performs an analysis, and can critique one). Analytics is where an AI assistant produces confident, wrong conclusions: Claude reports vanity metrics (page views, total signups), treats correlation as causation, reacts to a two-day spike or an n=20 sample as if it were a trend, quotes the average (which hides every interesting subgroup), and stops at "here are the numbers" instead of a decision. A dashboard nobody acts on, or an insight built on a bias, is worse than no analysis.

So this skill gates:

1. Establish context — the decision/question this informs, the goal it ladders to, and the data available (these set what to measure and which method).
2. Apply the always-true core — question first, actionable metrics, signal vs. noise, segment, quant+qual, end in a recommendation.
3. Surface the context-dependent decisions (analysis method, leading vs. lagging, tooling, attribution model, cohort type, retention window) with trade-offs; let the user choose.

Then it hands off to pm-okr-metric-validity-audit (are the chosen metrics valid, not vanity?) and pm-assumption-rigor-audit (do the causal claims and read-outs survive scrutiny?).

Scope: this skill owns the analysis process — what to measure, funnels/cohorts/segments/journeys, reading signal from noise, and the data-to-decision story. It defers the rigorous validity of a metric's definition to pm-okr-metric-validity-audit, the statistics of a controlled change (sample size, significance, guardrails) to pm-experimentation-ab, OKR/KPI artifacts to pm-okrs-kpis, and the discovery research that generates qualitative "why" to pm-discovery.

---

Step 0 — Establish context before analyzing

Ask if not known; state the assumption if proceeding without an answer:

• What decision or question does this inform? Start from the question, not the data — "why did activation drop?" or "which onboarding step loses users?" An analysis with no decision attached produces noise.
• What goal does it ladder to? Tie the metric to a business/user objective so you measure what matters, not what's convenient to track.
• What data exists, and is it trustworthy? Event coverage, time range, segments available, known tracking gaps. Thin or biased data changes what you can honestly conclude.

---

The always-apply core (true for any analysis)

Measure what matters

• Question first, then data. Follow the analytics lifecycle: set the measurement goal → collect → analyze → act. Each insight feeds the next question.
• Actionable over vanity metrics. Page views and total registered users flatter a deck but don't guide a decision. The test: "if this number moves significantly, do I know exactly what to do?" If not, it's a vanity metric. Prefer behavior-linked measures (first-week activation, feature adoption, retention).
• One primary metric + guardrails. Tie the analysis to one success metric and watch health/guardrail metrics so a "win" isn't paid for elsewhere. (The rigorous validity check of those metrics belongs to okr-metric-validity.)

Read the data honestly

• Separate signal from noise. Don't react to small or short-term fluctuations; use rolling windows (30/90/365-day), moving averages, and look for patterns that repeat across periods and segments before acting.
• Avoid the data traps — name them and design around them:
• Correlation ≠ causation — two things moving together (push notifications ↑ and usage ↑) may both be driven by a third (a holiday campaign). Look for a mechanism; confirm cause with an experiment (experimentation-ab).
• Survivorship bias — analyzing only the users who stayed hides why others left. Ask who's missing from the data.
• Sample-size seduction — a 50% lift on n=10 vanishes at n=1,000. Wait for enough data.
• Selection bias — decide success metrics before looking; document all analyses, not just the flattering ones.
• Averages lie — segment. The "typical user" usually doesn't exist. Break results down by device, new vs. returning, acquisition source, and behavior; the interesting story is almost always in a subgroup (overall +10% can hide mobile −5%).

Turn numbers into a decision

• Find the "why," then pair quant with qual. Numbers show what happened; talk to the users behind an interesting pattern (almost-converted, churned power users) to learn why — then return to the data to validate what you heard.
• End in insight → impact → action. Lead with the problem in business terms, not the raw metric ("new users can't find the core feature, ~$50K/mo" beats "DAU −15%"), choose a visualization that clarifies the one insight, and close with a specific, owned, measurable recommendation. An analysis that doesn't change a decision isn't done.

---

The context-dependent decisions (surface, don't auto-apply)

Present each with its trade-off and a recommendation tied to Step 0; let the user choose. Running every analysis type, or pulling every metric, is the failure mode.

| Decision | Apply when | Avoid / adapt when | Default recommendation | |---|---|---|---| | Analysis method | Funnel for step-by-step drop-off in a known flow; cohort for how behavior/retention evolves over time; segment for how groups differ; journey/path for non-linear, cross-touchpoint behavior | Forcing a funnel on exploratory behavior, or path analysis when a simple funnel answers it | Match the method to the question; funnel for conversion, cohort for retention, segment to explain variance | | Leading vs. lagging indicators | Leading (activation, feature adoption) to act early when the goal is rare/slow; lagging (MRR, LTV) to validate outcomes | Steering day-to-day on lagging metrics alone (too late to act) | Pair both — leading to guide action, lagging to confirm it worked | | Tooling / platform | GA for web traffic; Mixpanel/Amplitude for event/behavioral product analytics; Segment to unify pipelines; warehouse (BigQuery/Snowflake) for deep/historical; real-time for live monitoring | Over-engineering a warehouse pipeline for a question a product-analytics tool answers today | Use the event-analytics tool already in place; add a warehouse only when depth/history demands it | | Cohort type | Behavioral (did action X) to find the "aha moment"; time-based to isolate a release/seasonality; acquisition to compare channel quality | Reading one cohort type as if it answered all three questions | Behavioral cohorts to find activation drivers; time-based to measure a change's impact | | Retention window | N-day for daily-use products; weekly/monthly for less frequent; custom for natural cycles (Sunday meal-planning) | A daily-retention lens on a product used weekly (false alarm) | Match the window to the product's natural usage rhythm | | Attribution model | Last-click for simple conversion credit; linear/time-decay when multiple touchpoints contribute | Last-click when upstream channels start journeys that convert elsewhere (under-credits them) | Multi-touch (linear/time-decay) when the journey spans channels; last-click only for simple cases | | Qual depth | Light (support tickets, session notes) for a clear pattern; deep (interviews) when the "why" is load-bearing for a big decision | Acting on quant alone for a costly/irreversible call | Scale qual to the stakes; always sanity-check a surprising number against a user's words |

---

Validate the result (orchestration)

> Hand-offs name each lens by its installable skill name. Invoke one only if that skill is installed; if it isn't, this skill's own core already carries these rules — proceed without it rather than blocking.

After producing or revising, hand it to the audit lenses rather than declaring it done. These are candidate lenses — posture per docs/orchestration-policy.md, or route the whole artifact through pm-product-review. Here assumption-rigor is the always-relevant lens (auto-runs); the others are offered, tied to what the artifact actually contains. If the user invoked this skill for one specific thing, respect that scope.

• pm-assumption-rigor-audit (auto-runs) — do the causal claims hold (or are they correlation dressed as causation), is the read honest (not selection-biased), and are load-bearing conclusions evidenced rather than asserted?
• pm-okr-metric-validity-audit (offer — if it names/relies on metrics) — are the metrics actionable and valid (not vanity), correctly defined, and tied to the objective? This is the rigorous check this skill defers the metric-definition standard to.

Composition (per docs/orchestration-policy.md §9): for validating a change with statistical rigor (sample size, significance, guardrails, peeking), pm-experimentation-ab is downstream (offer to produce next, don't auto-generate); for a high-stakes, irreversible decision built on the analysis, pm-decision-quality-audit checks the decision process. If the audits surface a conflict, resolve toward the purpose: an analysis exists to drive a better decision — if it rests on a vanity metric, an unsegmented average, or an unproven causal claim, fix that before anyone acts on it.

---

Common do/don't patterns

| ❌ Don't | ✅ Do | |---|---| | Lead with the data | Start from the decision/question, then measure | | Report vanity metrics (page views, total signups) | Use actionable metrics — "if it moves, I know what to do" | | Treat correlation as causation | Look for a mechanism; confirm cause with an experiment | | React to a two-day spike or n=20 | Use rolling windows; wait for repeatable patterns / enough sample | | Analyze only the users who stayed | Ask who's missing — guard against survivorship bias | | Cherry-pick the flattering cut | Pre-commit success metrics; document all analyses | | Quote the average | Segment by device/source/behavior — averages hide the story | | Report what with no why | Pair quant with qual; validate one against the other | | Stop at the numbers | End with insight → impact → action, owned and measurable | | Reinvent A/B statistics here | Defer significance/guardrails to experimentation-ab | | Ship the analysis unchecked | Hand off to okr-metric-validity + assumption-rigor |

---

Source lessons (Uxcel)

• Analytics Strategy & Planning
• Analytics Tools & Platforms
• Customer Journey Analytics
• Digging into Data
• Analytics Storytelling
• User Funnel Analysis
• Cohort & Segment Analysis

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Uxcel-Lab
• Source: Uxcel-Lab/product-skills
• License: MIT
• Homepage: https://uxcel.com

Install and usage instructions live in the source repository linked above.