Ai Native Product

AI-Native Product Development

> "AI products aren't deterministic. They require continuous calibration, not just A/B tests."

This skill covers AI-Native Product Development — the overlay that modifies discovery, architecture, and delivery when AI is at the core. It addresses the unique challenges of building products where AI agents perform tasks autonomously.

Part of: Modern Product Operating Model — a collection of composable product skills.

Related skills: product-strategy, product-discovery, product-architecture, product-delivery, product-leadership

---

When to Use This Skill

Use this skill when:

• Building AI agents that act on behalf of users
• Adding LLM-powered features to existing products
• Designing human-AI interaction patterns
• Deciding how much autonomy to give AI
• Setting up eval strategies and calibration loops
• Managing the "agency-control tradeoff"

Not needed for: Traditional software products, ML models used only for backend optimization (no user-facing autonomy)

---

What Makes AI Products Different

Traditional Software vs. AI Products

| Dimension | Traditional Software | AI-Native Products | |-----------|---------------------|-------------------| | Behavior | Deterministic | Probabilistic | | Testing | Unit tests, QA | Evals, calibration | | Correctness | Binary (works or doesn't) | Spectrum (good enough?) | | User role | Operator | Delegator + Reviewer | | Failure mode | Error messages | Plausible but wrong outputs | | Iteration | Ship → Measure → Iterate | Ship → Observe → Calibrate | | Trust building | Feature completeness | Demonstrated reliability |

The Core Challenge

AI products must navigate a fundamental tension:

More autonomy = More value (fewer steps, faster outcomes) More autonomy = More risk (errors affect real work)

This is the Agency-Control Tradeoff.

---

Framework: The CCCD Loop

> Credit: Aishwarya Goel & Kiriti Gavini

AI products require a Continuous Calibration and Confidence Development (CCCD) loop:

┌─────────────────────────────────────────────────────────────────┐
│                        CCCD LOOP                                │
│                                                                 │
│    CALIBRATE → CONFIDENCE → CONTINUOUS DISCOVERY → CALIBRATE   │
│         ↓           ↓              ↓                 ↓         │
│     Eval and    Build user    Observe AI       Update evals    │
│     adjust AI    trust over   interactions     and models      │
│     behavior     time         at scale                         │
└─────────────────────────────────────────────────────────────────┘

CCCD Components:

| Component | Purpose | Activities | |-----------|---------|------------| | Calibrate | Tune AI behavior to match user expectations | Run evals, adjust prompts/models, set guardrails | | Confidence | Build appropriate user trust | Show AI reasoning, enable verification, demonstrate reliability | | Continuous Discovery | Observe AI-user interactions at scale | Log interactions, identify failure patterns, surface edge cases | | → Back to Calibrate | Update based on learnings | Improve evals, retrain, adjust prompts |

---

The Agency-Control Progression

Five Levels of AI Agency

| Level | Description | AI Does | User Does | Example | |-------|-------------|---------|-----------|---------| | 1. Assist | AI suggests, user executes | Generates options | Chooses and acts | Autocomplete, suggestions | | 2. Recommend | AI ranks, user approves | Analyzes and recommends | Reviews and approves | "AI recommends these 3 actions" | | 3. Execute with confirmation | AI acts after approval | Prepares action | Confirms before execution | "Send this email?" → Yes/No | | 4. Execute with notification | AI acts, notifies after | Acts autonomously | Reviews outcomes | "I scheduled the meeting and sent invites" | | 5. Fully autonomous | AI acts without notification | Handles end-to-end | Sets goals, reviews exceptions | AI handles routine tasks silently |

Progression Strategy

Start lower, earn higher:

Level 1 → Build trust → Level 2 → Demonstrate reliability → Level 3 → ...

Graduation Criteria:

| From Level | To Level | Requires | |------------|----------|----------| | 1 → 2 | Assist → Recommend | User accepts suggestions > 70% | | 2 → 3 | Recommend → Execute with confirm | User approves recommendations > 80% | | 3 → 4 | Execute+confirm → Execute+notify | User confirms without edit > 90% | | 4 → 5 | Execute+notify → Autonomous | User overrides 95% | | Factual accuracy | Is output factually correct? | > 98% | | Hallucination rate | Does AI make things up? | 80% | | Override rate | How often do users correct AI? | 80%, override rate < 15% | | Each GA increment | Metrics stable, no new failure modes |

Calibration Loop

Continuous calibration process:

OBSERVE → IDENTIFY → CALIBRATE → VALIDATE → DEPLOY
   ↑                                           │
   └───────────────────────────────────────────┘

| Step | Activities | Cadence | |------|------------|---------| | Observe | Monitor production interactions, logs, feedback | Continuous | | Identify | Surface failure patterns, edge cases, drift | Daily/weekly | | Calibrate | Adjust prompts, fine-tune, add guardrails | As needed | | Validate | Run evals on calibrated version | Before deploy | | Deploy | Ship updates, continue observing | Staged |

Calibration Triggers:

• Eval metrics below threshold
• New failure pattern identified
• User feedback trend (negative)
• Model update available
• New use case discovered

AI Metrics Hierarchy

LAGGING
├── User retention (AI users vs. non-AI users)
├── Task completion rate (with AI assist)
└── Revenue from AI features

CORE
├── User acceptance rate
├── Override rate
├── Time-to-completion (with AI)
└── User-reported satisfaction

LEADING
├── Eval metrics (accuracy, hallucination, etc.)
├── Interaction volume
├── Feature discovery rate
└── Feedback sentiment

GUARDRAILS
├── Harmful output rate
├── Latency P95
├── Error rate
└── Cost per interaction

---

AI-Specific Anti-Patterns

| Anti-Pattern | Why It Fails | Instead | |--------------|--------------|---------| | Ship and hope | AI behavior drifts without monitoring | Continuous calibration | | Autonomous by default | Users don't trust, don't adopt | Earn autonomy progressively | | Black box AI | Users can't verify, won't trust | Show reasoning, enable verification | | No evals | Quality degrades silently | Comprehensive eval strategy | | Ignore overrides | Miss calibration signals | Override patterns inform calibration | | One-size-fits-all agency | Different tasks need different levels | Task-specific agency levels |

---

Templates

This skill includes templates in the templates/ directory:

• agency-assessment.md — Determine appropriate agency level
• eval-strategy.md — Design eval suite for AI feature
• calibration-plan.md — Set up continuous calibration

---

Using This Skill with Claude

Ask Claude to:

1. Assess agency level: "What agency level should [AI feature] have?"
2. Design agency progression: "Create a graduation path from assist to autonomous for [feature]"
3. Identify failure modes: "What could go wrong with [AI feature]? How do we mitigate?"
4. Design eval strategy: "Design an eval suite for [AI feature]"
5. Plan calibration: "Create a calibration plan for [AI feature]"
6. Adapt discovery: "What AI-specific questions should I ask in discovery for [use case]?"
7. Design confidence building: "How should [AI feature] show its reasoning?"
8. Plan AI rollout: "Create a staged rollout plan for [AI feature]"
9. Set AI metrics: "What metrics should we track for [AI feature]?"
10. Review AI brief: "Critique this solution brief for AI considerations"

---

Connection to Other Skills

| When you need to... | Use skill | |---------------------|-----------| | Define overall product strategy | product-strategy | | Run discovery (with AI adaptations) | product-discovery | | Structure bets and roadmap | product-architecture | | Plan rollout and metrics | product-delivery | | Scale AI products across teams | product-leadership |

---

Quick Reference: AI Product Checklist

Before shipping AI features:

• [ ] Agency level defined — Clear level for this feature
• [ ] Graduation criteria set — How we'll earn higher autonomy
• [ ] Failure modes mapped — Know what can go wrong
• [ ] Evals in place — Automated quality checks
• [ ] Human evals scheduled — Subjective quality review
• [ ] Calibration loop running — Continuous improvement process
• [ ] Confidence mechanisms built — Users can verify AI work
• [ ] Guardrails active — Prevent harmful outputs
• [ ] Rollout staged — More cautious than traditional features
• [ ] Override tracking — Learning from user corrections

---

Sources & Influences

• Aishwarya Goel & Kiriti Gavini — CCCD Loop, Agency-Control Trade-off
• Anthropic — Constitutional AI, RLHF approaches
• OpenAI — Eval best practices
• Google DeepMind — AI safety frameworks

---

Part of the Modern Product Operating Model by Yannick Maurice

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: yannickYamo
• Source: yannickYamo/skills
• License: MIT

Install and usage instructions live in the source repository linked above.