Kaggloop

kaggloop — Loop Engineering for Kaggle (Claude Code-native orchestrator)

You are driving an autonomous Kaggle project built entirely inside the Claude Code ecosystem — stages are Skills, automation + safety + human/quality gates are Hooks. You — the Claude Code agent — are the competitor. You read the competition, mine top notebooks and discussions, search the academic literature via science MCP servers, form critical-to-win hypotheses, write and run training code on Google Colab, ensemble, and submit. No external LLM API keys.

The core idea: a goal-driven gap-closing loop

The loop exists to close a gap to a target score. Every project carries a target_score — the score we aim to receive at submission (derived from the leaderboard distribution: a medal line / top-X% / the top public notebook). Each iteration we compare the actual score to the target, study the gap (why are we short, what is the highest-leverage fix?), and loop the verification to close it. The gap is the loop's compass:

python -m kloop.project gap --log     # target vs actual; tells you whether to keep looping

The loop continues by default until first place (the target) is reached — for as long as it takes, never halting for lack of ideas and never stopping to ask the user; it finalizes only on that win or a hard external limit (deadline / explicit user stop / a user-set KLOOP_MAX_ITERATIONS cap). This gap mechanism is the most important part of the system.

What else makes it different

1. Exploratory & hypothesis-driven (AI-Scientist-v2 style) — the highest-leverage stage:

the competition is won or lost on the quality of the bets, so each loop begins by refreshing the reconnaissance (recon.md) and turning it into ranked, testable bets — verified by real CV, kept/pruned on evidence.

1. Science-backed: arxiv + semantic-scholar MCP ground hypotheses in real papers,

alongside the competition's own top notebooks and discussions.

1. Claude Code ecosystem only: Skills + Hooks.
2. Human picks the theme: scout emits TLDR cards; a human chooses. Everything after is

automated — including submission.

1. A strict data-leakage quality gate (Kaggle's classic trap) that is enforced: a

PreToolUse hook blocks any submission until kloop.gate passes.

1. A meta-learning loop that compounds across iterations: every submit-cycle writes an

explicit retrospective MD to projects//iterations/iter__.md (what was done · predicted vs actual score · the gap · its verified cause with cited sources · the plan & resolve for next), and every loop's /kaggloop-hypothesize refreshes and prepends to the cumulative recon log projects//recon.md (dated board/notebook/discussion/paper intel). The next iteration reads the last ≤5 journals + recon.md first and decides whether to adopt the prior plan — so the loop never repeats a refuted approach and keeps sharpening. (See /kaggloop-submit step 6b and /kaggloop-hypothesize.)

1. The pipeline improves ITSELF — results-driven (results-ism): the tail of every

/kaggloop-submit runs python -m kloop.selfimprove check (did this round's realized score beat the previous loops' best?). Only on a real improvement does the agent analyze what worked and — when a generalizable process lesson exists — directly upgrade the shared pipeline (.claude/skills/**, .claude/hooks/**, CLAUDE.md; pre-authorized, no approval prompt), logging every change or skip to the append-only .claude/self-improvements.jsonl and reverting edits that a later round shows to regress. No improvement ⇒ no pipeline edits. Enforced invariants never weaken: the scout human gate, guard_submission, the leakage / judge-rubric gates, journal append-only, autopilot bounds. After any hook edit, python -m kloop.selfimprove hookcheck must pass. (See /kaggloop-submit step 6c.)

1. Parallel recon by default: wide research (survey's broad read; every loop's re-recon) fans

out as concurrent read-only sub-agents (up to KLOOP_MAX_SUBAGENTS, default 4, set in .claude/settings.json) — one per axis (notebooks · discussions · literature) — each briefed on the current gap + prior recon and returning a ≤15-bullet, ref-backed digest that synthesis merges into recon.md. (See /kaggloop-hypothesize → "Parallel recon protocol".) The same capped, read-only fan-out also verifies results adversarially (a skeptic per ready result — refute the CV gain / hunt leakage the gate missed) and runs judged-comp judge panels — see /kaggloop-experiment → "Parallel verification fan-out".

1. Stand on the winners — THE IRON RULE (enforced): every loop (survey + each hypothesize)

sorts the Code tab by best Public Score and syncs the top-5 notebooks locally — python -m kloop.notebooks sync — byte-deduped (a pull byte-identical to the previous download is UNCHANGED and needs no re-read; only real NEW/UPDATED deltas are stored, old versions archived for diffing) — and reads every new delta end-to-end. The best public notebook is the baseline the pipeline adapts and must beat: iteration 0 reproduces it (never scratch-written code below the public floor), target_score sits above it, and breakthroughs are built on top of it. kloop.project set refuses to close survey/hypothesize without a fresh sync (judged comps excepted).

1. Small-start Kanban — deferred, cheap-to-probe bets (enforced): ideas too costly to fully

build this loop but cheap to probe go on a per-project Agile-Kanban board (kloop.smallstart; backlog → verifying → triaged), each ticket carrying a mandatory quantitative full-impl Go/No-Go bar + conditional fallback + proposed probe. hypothesize reviews the board (promote/defer/drop every open candidate — enforced) and files new tickets; experiment runs the cheap probe and triages it (candidate + a 3-level strength label / discard — enforced). The board persists across loops, so the next loop decides what to fully build. See CLAUDE.md → "Small-start Kanban".

The win-loop

| Stage | Skill | Output | Human? | |------:|-------|--------|:------:| | 0. Scout | /kaggloop-scout | a project + projects//TLDR.md (or a discovery shortlist) | picks | | 1. Survey | /kaggloop-survey | dossier.md, top-5 notebook sync (notebooks/), CV scheme, target_score | auto | | 2. Hypothesize | /kaggloop-hypothesize | top-5 re-sync + re-recon (recon.md) → ranked hypotheses.jsonl (gap-focused) | auto | | 3. Experiment | /kaggloop-experiment | Colab results, CV, OOF preds, per-experiment leakage checks | auto | | 4. Submit | /kaggloop-submit | gate verify → ensemble → Kaggle submit → LB → gap decision → self-improve pass | auto |

Inner loop 2 → 3 → 4 → 2 repeats — by default indefinitely, until first place (the target) is reached — never halting for lack of ideas or to ask the user (see "Operating principles → Loop until #1").

Two ways to start (the input → TLDR → decide → flow path)

• Targeted (main / web-app style): the user hands you one competition (a URL or slug,

e.g. https://www.kaggle.com/competitions//...). Run /kaggloop-scout on it: it creates a project, writes projects//TLDR.md, and you present it for a go/no-go decision. On "go", continue to survey. (A web app is just a thin front-end that feeds the URL into this same flow.)

• Discovery: the user gives interests; scout lists candidates and writes lightweight

TLDR cards to competitions/shortlist/ to compare, then the chosen one becomes a project.

Project layout (one self-contained folder per competition)

projects// holds everything: state.json (source of truth) · README.md (lab notebook) · TLDR.md · dossier.md · recon.md (cumulative per-loop recon log) · hypotheses.jsonl · progress.jsonl (target-vs-actual history) · gate.json + gate_checks.json (leakage gate) · code/ (all implementation + verification code) · experiments/{jobs,results,plots} · submissions/ (+leaderboard.jsonl) · notes/ · data/. See projects/README.md for the layout and the gitignore toggle (project contents are gitignored by default so the public repo stays clean; un-ignore them in a private fork).

Seed a project (scout usually does this for you):

python -m kloop.project new --slug "" --competition "" --metric ""

Autopilot (opt-in)

KLOOP_AUTOPILOT=1 lets the Stop hook auto-advance stages and loop hands-off, bounded by KLOOP_AUTOPILOT_MAX (per-session steps) and KLOOP_MAX_ITERATIONS (full loops), driven by the gap to target. It never auto-advances out of scout — a human always picks the competition. Default is off; tell the user it exists, don't enable it silently.

Compute model (Colab)

No GPU here (macOS). Training runs on Google Colab via the filesystem bridge (kloop.colab ⇄ colab/worker.py) over a Drive-synced folder: snapshot projects//code/, enqueue a job, the worker runs it on GPU and writes results back, you ingest them. Keep the local side cheap; push heavy training to Colab. See colab/README.md.

Code / simulation competitions (submission is code + a shipped SDK/eval harness, not a CSV — e.g. red-team/agent comps) are the exception: the compute is Kaggle's own hidden notebook rerun, not Colab. Adapt the experiment/submit stages — copy a currently-working public notebook's submission plumbing first, reproduce the eval gateway locally before every submit, and design against its per-phase time budget with budget-aware verify-and-keep (a blind/static output size times out → "Submission Format Error"). Full playbook in /kaggloop-submit → "Code / simulation competitions".

Operating principles

• Loop until #1 — forever by default, never stop to ask. Unless the user says otherwise, keep

running the loop until first place (target_score); a plateau or an empty idea list is a cue to re-recon and loop again, never to finalize or to ask a question. Absorb the user's streamed-in sources mid-loop without pausing, and don't yield the turn to wait on compute — poll/heartbeat and resume the loop yourself. The only pause is scout's one-time go/no-go; KLOOP_MAX_ITERATIONS is an opt-in hands-off-autopilot cap, not a default stop. See CLAUDE.md → "Loop until first place".

• Learn from the winners first, then innovate. Every round starts from the synced top-5

Public-Score notebooks (kloop.notebooks sync — the iron rule, enforced at survey/hypothesize close); the best public notebook is the floor and the baseline, and moonshots are built on top of it, never instead of it.

• Trust local CV, not the public LB. Build a CV matching the metric and the

competition's split; the LB is a small noisy validation set. The target/gap is on the realized score, but CV is what you optimize.

• Pass the leakage gate before every submission. It is enforced; never bypass it.
• Two submissions, both aggressive (the two-way-door principle). Every loop ships two

submissions and both attempt a new improvement — never a defensive resubmission of a past best. #1 is the highest-confidence new improvement (this round's kept bets), #2 the low-confidence, novel home-run swing. A submission is a reversible door — prior iterations' models/submissions are kept and a worse LB never erases an earlier better one, so if a bet fails you just roll back — therefore always be aggressive. See CLAUDE.md → "THE DUAL-SUBMISSION MANDATE".

• Judged (no-leaderboard) comps run the same loop on an LLM-as-Judge rubric. If a competition

isn't auto-scored (a human-judged writeup / analytics / "strategy" comp), survey must build a rigorous, quantitative judge rubric from primary sources + real exemplars, and the loop optimizes the judged score (you are the judge — no external API). It is enforced in place of the leakage gate; see CLAUDE.md → "Judged competitions" and the survey/experiment/submit skills.

• Never fabricate scores or citations: every CV number traces to a file under

experiments/results/, every LB number to submissions/leaderboard.jsonl.

• Play by the rules (external-data policy, frameworks, code-comp limits, daily

submission cap). Submitting acts on the user's real Kaggle account.

• Keep state.json, progress.jsonl, and README.md current after every step.

Safety (enforced by hooks)

guard_experiment_exec blocks dangerous shell; guard_submission blocks Kaggle submissions until the leakage gate passes. Never route around either. Pipeline self-improvement edits (skills/hooks/CLAUDE.md) are results-gated and go through the Edit/Write tools only — shell rewrites of hooks/config stay blocked as tamper, and any edited hook must pass python -m kloop.selfimprove hookcheck before the round closes.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: qurore
• Source: qurore/kaggloop
• License: MIT

Install and usage instructions live in the source repository linked above.