Finfluencer Audit

Finfluencer Audit

Most "this guru has a 70% win rate" claims collapse under two questions: win rate on what fraction of their content? and versus what baseline? This skill answers both, reproducibly.

Non-negotiables

1. Freeze before you score. Write the sample list and the claim ledger to

disk before loading a single price. Otherwise you will unconsciously read ambiguous headlines in the direction the market went.

1. Always report a baseline. A hit rate without always-long next to it is

marketing, not evidence. In a bull market, "just hold it" often beats the guru.

1. Report the falsifiable rate. If only 40% of videos contain a checkable

claim, say so in the headline. Accuracy on a self-selected 40% is not accuracy.

1. Right-censor, don't guess. A 12-month call made 3 months ago is

unresolved, not wrong.

1. Grade the evidence tier. Coding from titles is much weaker than coding

from transcripts. Never let the reader assume you had transcripts.

1. No accusations. Report what was said and what the market did. Flag

unverifiable credentials as unverified, not as false.

Pipeline

fetch_archive → build_sample → fetch_content → code_claims
                                                     ↓  (freeze)
                                   report ← backtest ← fetch_market

Each step writes JSON into a run directory so any number can be traced back to a video URL.

1. Build the sampling frame

python3 scripts/fetch_archive.py --platform bilibili --uid 25270495 \
  --since 2023-08-06 --until 2026-08-06 --out runs/

Bilibili's space archive needs a WBI signature, dm_img_* anti-bot fields, and buvid cookies all at once — see [references/platforms.md](references/platforms.md) for the failure codes and what each one means. For YouTube use --platform ytdlp --channel-url ....

2. Draw a pre-registered sample

python3 scripts/build_sample.py --run runs/ --margin 0.08 --seed 20260806 --two-phase

Sample size is Cochran's formula with a finite-population correction. Use --two-phase when most titles are chit-chat: it screens for directional language and oversamples that stratum, which is the difference between 42 scoreable claims and ~90 from the same budget.

Publish sample.json before continuing. Substitute only from alternates.json, and only for deleted or unreadable videos.

3. Collect evidence

python3 scripts/fetch_content.py --run runs/ --transcribe whisper --transcribe-limit 30

Subtitles first, audio transcription second, title-only last. Check content_meta.json — if subtitle + transcript coverage is near zero, your audit is a headline audit and the report must say so.

4. Code claims, then freeze

python3 scripts/code_claims.py --run runs/ --dual-channel

Direction is resolved per asset per clause, because titles like 「原油暴跌，黄金后市可期」 are bearish oil and bullish gold. --dual-channel runs a second stricter coder and reports Cohen's kappa; below 0.6, say plainly that coding noise rivals the measured effect.

Coding rules and the claim taxonomy live in [references/coding-protocol.md](references/coding-protocol.md).

5. Get prices

python3 scripts/fetch_market.py --run runs/ --start 2023-07-01 --end 2026-08-07

Every free source fails sometimes; the script walks Yahoo → Stooq → FRED and records which one answered. See [references/market-data.md](references/market-data.md).

6. Backtest

python3 scripts/backtest.py --run runs/ --as-of 2026-08-06

Entry is the first close on or after the publish date. Outputs hit rate with a Wilson interval, signed returns, MFE/MAE, and three baselines, cut by horizon, asset, direction, and claim type.

7. Report

python3 scripts/report.py --run runs/ --name "BOSS墨" --profile-url https://space.bilibili.com/25270495/

Writes report.md and canvas_payload.json. For an interactive deliverable, render the payload with the canvas skill.

Beyond price accuracy

A reliability verdict needs more than a hit rate. Also check, and cite sources for each:

• Credentials — is the self-described qualification independently findable

(regulator registry, patent/trademark office, employer)? Record verified, not_found, or unverified_hearsay — never false without documents.

• Track record — brokerage statements or third-party audit, or hearsay?
• Conflicts — paid courses, private groups, referral links, token bags.
• Error handling — do they publish corrections, or only "as planned" recaps?

A high post_hoc_review count with a low ex_ante_prediction count is a tell.

• Selection — do they delete losing calls? Compare archive count against the

platform's reported total.

Pitfalls

Read [references/pitfalls.md](references/pitfalls.md) before your first run. The short version:

| Trap | Consequence | | --- | --- | | Whole-title direction voting | Multi-asset calls get inverted | | No baseline | Bull-market drift reads as skill | | Counting unmatured calls as misses | Recent calls drag the score down | | Title-only coding presented as analysis | Overstated confidence | | Sampling without a seed | Nobody can reproduce you | | Scoring before freezing the ledger | Hindsight leaks into coding |

Output contract

A finished audit answers, in this order:

1. What fraction of content was checkable at all?
2. Of that, what was the hit rate, with a confidence interval?
3. How does it compare to always-long, coin flip, and trend following?
4. Which specific calls hit and missed, with links?
5. What is unverifiable about the creator's own claims?
6. What would change the verdict?

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Faust-Donf
• Source: Faust-Donf/finfluencer-audit
• License: MIT

Install and usage instructions live in the source repository linked above.