Thinking Probabilistic

Probabilistic Thinking

Overview

Probabilistic thinking, informed by Philip Tetlock's "Superforecasting," treats a forecast as a probability and a range rather than a single confident number. Three moves do almost all the work: anchor on the base rate, express the estimate as a range (not a point), and update the number when new evidence arrives.

Core Principle: Start from how often similar things happen, state your estimate as a range with a confidence level, and move the number — explicitly — when the evidence moves.

> Stateless-agent note. Across a single task you have no persistent prediction log, so there is no "track my calibration over months" step here. The leverage is in the act of estimating: base rate, range, update. Apply the calibration attitude (assume you're overconfident; widen the range) without pretending to keep a cross-session scorecard you don't have.

When to Use

• Stating a timeline or effort estimate
• Assessing the risk of an action (migration, deploy, change)
• Predicting an outcome (will this fix work? will this launch hit the target?)
• Evaluating an uncertain technical choice
• Any time you're about to give a confident single number you can't actually be sure of

Decision flow:

About to state a forecast/estimate/risk?
  → Outcome genuinely uncertain? → yes → BASE RATE, then a RANGE (not a point)
  → New evidence since last estimate? → yes → UPDATE THE NUMBER
  → Can you just look it up / measure it? → yes → DO THAT INSTEAD

When NOT to Use

• The quantity is knowable. If you can measure it, query it, or look it up, do that — don't dress a checkable fact as a probability.
• A single piece of evidence updates a single prior. That's the narrower mechanics of thinking-bayesian; use it for the explicit prior × likelihood-ratio update.
• The decision doesn't depend on the number. If you'd act the same across the plausible range, skip the estimate and act.
• You'd be inventing the base rate. If there's no real reference class, say the estimate is a guess rather than manufacturing false precision.

Core Concepts

Probability as Confidence

Convert vague language to numbers:

| Vague Statement | Probability Range | |-----------------|-------------------| | "Certain" | 99%+ | | "Almost certain" | 90-99% | | "Very likely" | 80-90% | | "Likely" / "Probable" | 65-80% | | "Better than even" | 55-65% | | "Toss-up" | 45-55% | | "Unlikely" | 20-35% | | "Very unlikely" | 10-20% | | "Almost impossible" | 1-10% | | "Impossible" | These are sanity checks you apply now, within the task — not a longitudinal tracking exercise.

The Equivalent Bet Test

"Would I bet at these odds?"

Prediction: 80% confident project finishes on time
Equivalent: Would I bet $4 to win $1?
If that feels wrong, adjust the probability.

The Outside View

Always check base rates:

Inside view: "Our team is great, we'll definitely finish on time"
Outside view: "What % of similar projects finished on time?"

Inside tends toward overconfidence
Outside provides calibration anchor

The Pre-Mortem Adjustment

Imagine failure, then adjust:

Initial estimate: 85% success
After pre-mortem: Identified 5 failure modes I hadn't considered
Adjusted estimate: 70%

The Confidence Interval Check

Are your intervals too narrow?

Test: Of your 90% confidence intervals, do 90% contain the actual?
Common finding: Only 60-70% do
Fix: Widen intervals by 50%

Application Examples

Project Estimation

## Project: Payment System Rewrite

Timeline estimate:
- 50% confidence: 8-12 weeks
- 80% confidence: 6-16 weeks
- 95% confidence: 4-24 weeks

Key variables:
- API complexity: High uncertainty (+/- 3 weeks)
- Team availability: Medium uncertainty (+/- 2 weeks)
- Integration testing: High uncertainty (+/- 4 weeks)

Commitment: "We're 80% confident we'll deliver in Q2"

Risk Assessment

## Risk: Database migration causes extended downtime

Probability assessment:
- Base rate for similar migrations: 20% have issues
- Our preparation level: Above average (-5%)
- Complexity of our schema: Above average (+5%)
- Rollback plan quality: Strong (-5%)

Estimate: 15% probability of extended downtime

Mitigation value:
- If issue occurs: 4 hours downtime × $10K/hour = $40K
- Expected loss: 15% × $40K = $6K
- Mitigation cost: $3K for additional testing
- Decision: Mitigation worth it (ROI positive)

Technical Decision

## Decision: Adopt new framework

Success probability factors:
| Factor | Probability | Weight |
|--------|-------------|--------|
| Team learns quickly | 70% | 0.3 |
| Framework matures | 80% | 0.2 |
| Performance meets needs | 60% | 0.3 |
| Integration works | 75% | 0.2 |

Combined probability (simplified):
0.70 × 0.80 × 0.60 × 0.75 = 25% (if all must succeed)
OR weighted average: 70% (if partial success acceptable)

Decision: High uncertainty suggests pilot first

Probabilistic Thinking Template

# Probabilistic Assessment: [Prediction]

## Prediction
[Clear, falsifiable statement with timeframe]

## Initial Probability
Estimate: [X]%
Base rate: [Similar events: Y%]
Adjustment rationale: [Why different from base rate]

## Confidence Interval
- 50% CI: [Range]
- 80% CI: [Range]
- 95% CI: [Range]

## Key Uncertainties
| Uncertainty | If positive | If negative |
|-------------|-------------|-------------|
| [Factor 1] | +X% | -Y% |
| [Factor 2] | +X% | -Y% |

## Updates (within this task)
| New information | Old P | New P |
|-----------------|-------|-------|
| | | |

## Checkable Outcome
[The specific observation that will prove this forecast right or wrong]

Verification Checklist

• [ ] Expressed prediction as specific probability
• [ ] Checked base rate for similar events
• [ ] Created appropriate confidence intervals
• [ ] Identified key uncertainties and their impacts
• [ ] Stated the prediction so it's checkable (claim + timeframe + range)
• [ ] Applied equivalent bet test for sanity check
• [ ] Willing to update the number when new information arrives

Key Questions

• "What probability would I assign to this?"
• "What's the base rate for similar things?"
• "What would change my estimate up or down?"
• "Am I being overconfident? (Usually yes — widen the range)"
• "Have I given a range, or am I hiding uncertainty behind a single number?"
• "Would I bet at these odds?"

Tetlock's Superforecaster Traits

1. Update often: Change the number when evidence changes
2. Granular probabilities: Use 65% not "likely"
3. Outside view: Start with base rates
4. Seek disconfirming evidence: Look for reasons you're wrong
5. Ranges, not points: Express confidence as an interval, and widen it
6. Intellectual humility: Assume you're often wrong

Tetlock's Wisdom

"The fox knows many things, but the hedgehog knows one big thing."

Superforecasters are foxes—they integrate many perspectives, update frequently, and avoid ideological certainty. They're not smarter; they're more calibrated.

"Beliefs are hypotheses to be tested, not treasures to be protected."

Your predictions should change as evidence changes. Holding steady when you should update is a calibration failure.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: tjboudreaux
• Source: tjboudreaux/cc-thinking-skills
• License: MIT

Install and usage instructions live in the source repository linked above.