Kaggloop Submit

Stage 4 — Submit (gate → ensemble → submit → study the gap → decide)

Convert this round's kept models into the strongest valid submission, pass the leakage gate (the submission guard will block you otherwise), push to Kaggle, record what the leaderboard says, then compare to the target and study the gap to decide the next move.

Both of this round's two submissions must attempt a NEW improvement — never a defensive resubmission of a past best (the two-way-door principle). Submission #1 (primary) is this loop's highest-confidence new improvement — the kept, gap-closing bets applied on top of the current best, a genuine measured step forward, not a prior iteration's model re-submitted to guard the score. Submission #2 (challenge) is the low-confidence, high-variance home-run swing (step 5b). Because a submission is a reversible door — every earlier iteration's models and submissions are kept, Kaggle holds two final-selection slots, and a worse LB this round never erases a better earlier one — a bet that fails just means you roll back and restart from the previous iteration. The downside is bounded and undoable, so always be aggressive: neither submission may be a score-protecting rehash.

Preconditions

• kept hypotheses with OOF + test predictions under experiments/results/.

python -m kloop.project set --stage submit --status running. A submission-format reference from the dossier.

Judged competitions (no CSV — the judge-rubric gate, enforced) — READ FIRST

If survey set scoring_mode = judged / hybrid (a human-scored Writeup; no leaderboard), the tabular flow below (ensemble → leakage gate → kaggle submit CSV) does not apply. Instead:

1. Assemble the final deliverable — the ≤word-limit Kaggle Writeup + its required

attachments (media/figures, and for an agent challenge the agent/deck it documents), matching the Submission Requirements exactly.

1. Judge-rubric gate (enforced — replaces the leakage gate). Run a final blind, adversarial

judging pass over the assembled deliverable against judge_rubric.json; write judge/iter_.json (per-sub-criterion scores + quoted evidence + weighted total). Record the realized score and study the gap on it: ``bash python -m kloop.project set --best-lb --best-submission \ --note "iter judged=/100" python -m kloop.project gap --log # target(rubric) vs judged actual — the compass python -m kloop.journal log --kind gate --decision "judge-rubric gate: judged=/100" \ --rationale "scored vs fixed anchors; weakest crit =; evidence in judge/iter_.json" ` **Do not finalize** without a primary-source judgerubric.json **and** a fresh judge/iter.json for this iteration — that is the enforced quality gate for judged comps. 2b. **The challenge deliverable (the dual-submission mandate, judged form).** Judged rounds ship a challenge too: the challenge-track bet becomes a **bold, interdisciplinary variant or extension of the deliverable** (a section/demo/artifact importing a mechanism from a foreign field). Judge it blind against the same rubric, write judge/iter__challenge.json, and journal --kind challengesubmission (or --kind challengedeferred with the hard blocker) — the stage will not close without one of the two. The better-scoring variant becomes the writeup candidate and --best-lb` carries its total.

1. The human submits the Writeup on the Kaggle website (New Writeup → attach assets → pick a

Track → Submit) before the deadline — the one manual gate (like scout). There is no kaggle submit CSV and guard_submission does not fire; never fabricate a leaderboard number for a judged comp.

1. Then continue at 6b/6c/7 below — write the mandatory iteration journal (predicted vs

judged, the per-criterion gap + its cause with cited sources, next plan), run the results-driven self-improvement pass (the judged rubric total recorded via --best-lb feeds kloop.selfimprove check exactly like a leaderboard score), and make the loop decision on the judged gap: by default loop back to hypothesize on the weakest-weighted criteria (never stop to ask), finalizing only when the target rubric score is met or a hard external limit is hit (deadline / explicit user stop / user-set KLOOP_MAX_ITERATIONS cap).

Procedure

1. Assemble OOF/test predictions from the kept models (ingest any pending Colab results

first: python -m kloop.colab ingest).

1. Build the ensemble, validated on the same dossier CV — the blend must beat the

best single model on local CV, or ship the single model: ``bash python -m kloop.score blend results//artifacts/oof.npy results//artifacts/oof.npy \ --weights 0.6 0.4 --out experiments/blend_oof.npy --metric --y-true code/y_true.npy ` For >2 models prefer kloop.score.greedyblend (Caruana-style). Apply the chosen weights to the **test** predictions to make the submission CSV. Sanity-check it against samplesubmission (columns, row count, id order, ranges, no NaNs). Journal it: `bash python -m kloop.journal log --kind ensemble --decision "blend h0003*0.6 + h0007*0.4" \ --rationale "blend CV 0.849 > best single 0.846; decorrelated" --evidence "experiments/blend_oof.npy" ``

1. Pass the leakage gate (mandatory — the guard enforces it). Run the automated checks

on the final ensemble, affirm the checklist, and verify: ``bash python -m kloop.gate check --task --train-ids code/train_ids.npy --test-ids code/test_ids.npy \ --oof experiments/blend_oof.npy --y-true code/y_true.npy --x code/X.npy \ --groups code/groups.npy --folds code/folds.npy python -m kloop.gate affirm --confirm fit_on_train_only,oof_target_encoding,no_future_info,no_banned_external,cv_matches_split,no_test_in_train python -m kloop.gate verify # writes gate.json passed:true; without it, submit is BLOCKED python -m kloop.journal log --kind gate --decision "leakage gate passed" --rationale "no fails; checklist affirmed" ``

1. Respect submission budget. The daily cap lives in project state

(max_daily_submissions — set during survey via python -m kloop.kaggle limits --save); count today's used slots with python -m kloop.kaggle submissions . A round ships two submissions — the main ensemble (step 5) plus the challenge submission (step 5b) — so it wants two remaining slots; the main sub always goes first when the budget is tight. Keep two final-selection slots in mind for the competition's end.

1. Submit and log it:

``bash python -m kloop.kaggle submit -f submissions/.csv -m "iter: blend cv=" \ --watch # polls until scored; prints scoring_seconds python -m kloop.project set --best-lb --best-submission submissions/.csv \ --note "iter LB= (cv=)" python -m kloop.journal log --kind submission --decision "submitted .csv" \ --rationale "cv=, expected ~target" --evidence "submissions/leaderboard.jsonl" ` (Append {file, cv, lb, message, track: "standard", scoringseconds, ts} to submissions/leaderboard.jsonl. scoringseconds — how long Kaggle took to score the submission — comes from the --watch output (or python -m kloop.kaggle watch after the fact); poll-interval accuracy is fine, and null` when the transition wasn't observed — e.g. a code-comp rerun scored overnight.)

5b. The challenge submission (mandatory second submission — enforced at stage close). Every round also ships the challenge-track artifact verified in /kaggloop-experiment (the interdisciplinary breakthrough bet — kloop.ledger list marks it CH). After the main submission is in, re-run the leakage gate on the challenge artifacts (gate check on its OOF/preds → affirm → verify — it is a different model; its gate run must be its own), then submit and journal it: ``bash python -m kloop.kaggle submit -f submissions/_challenge.csv \ -m "iter CHALLENGE: cv=" --watch # prints scoring_seconds too python -m kloop.journal log --kind challenge_submission \ --decision "challenge sub LB= (main LB=)" \ --rationale "" \ --evidence "submissions/leaderboard.jsonl" ` Append it to submissions/leaderboard.jsonl with track: "challenge" (+ its scoringseconds, as in step 5). --best-lb / --best-submission take **whichever of the two submissions scored better** — when the challenge sub wins, the leapfrog worked: promote it to next round's standard baseline. Only a **hard blocker** — zero remaining daily submissions, a gate-failing / structurally broken challenge artifact (rejected in the ledger with the reason), the deadline — may skip it, and that skip must be journaled honestly: `bash python -m kloop.journal log --kind challenge_deferred --rationale "" \ --decision "challenge submission deferred" ` kloop.project set **refuses to close this stage without one of the two records** (challengesubmission or challenge_deferred`) for this iteration — never game the deferral: "CV was worse than the main sub" is NOT a blocker (upside variance is the point).

1. Study the gap (the core of the loop). Compare actual to target and analyze why:

``bash python -m kloop.project gap --log # appends target vs actual to progress.jsonl python -m kloop.standing snapshot --note "iter: " # append our score vs the live # medal landscape (top score + gold/silver/bronze cutoff scores, our rank & medal) to # projects//standing.jsonl — one stacked record per iteration. Pass --name when # multiple projects run concurrently (current_project is a shared cache). ` Read the standing: how far is our realized score from the **bronze/silver/gold lines** and from top? Did LB move with CV? Is the gap from underfitting, a CV↔LB mismatch (shake-up / leakage / distribution shift), or a metric/post-processing miss? **Compare the two tracks:** did the challenge submission beat the main one (leapfrog → promote it to next round's baseline), land close (the mechanism has signal → sharpen it next round), or crater (retire it and pick a fresh challenge axis)? **And check the public floor:** is our realized score still below the best public notebook's Public Score (the synced top-5 — python -m kloop.notebooks list + recon.md)? Below the floor, the gap analysis must explain *why we underperform code anyone can fork*, and closing to that baseline (adapt it — it's local) is next loop's #1 bet before any exotic idea. **Never overfit to the public LB** — trust CV. Journal the analysis (required to close the stage): `bash python -m kloop.journal log --kind gap_analysis \ --decision "gap remains; likely " --rationale "" ``

6b. Write the iteration learning journal (MANDATORY — the meta-learning loop). Every iteration must leave an explicit, human-readable retrospective MD at projects//iterations/iter__.md (zero-padded, one per submit-cycle). The next iteration's /kaggloop-hypothesize reads the last ≤5 of these first, so write it to be useful to your future self: honest, specific, and grounded. Required sections (in order): ``markdown # iter — · · version/sub: · LB: ## What was done # the approach + exact config/knobs changed vs last iter (be concrete) ## Predicted score # the number you expected BEFORE submitting + how you derived it ## Actual score # the real LB (traceable to leaderboard.jsonl / Kaggle); "blank COMPLETE"⇒verify it's not a failure ## Gap # predicted−actual AND target−actual; was the prediction right? ## Gap investigation # WHY the gap — verified against real resources: the synced top-5 # notebooks (projects//notebooks/), discussions, the science # MCP (arxiv/semantic-scholar), the SDK source, a local harness # repro. Cite each. No hand-waving. Include: above or below the # best public notebook's score, and why. ## Challenge track # the 2nd (challenge) submission: the bet, its LB vs the main sub, # verdict (leapfrog → new baseline / signal → sharpen / retire); # or the journaled challenge_deferred hard blocker ## Next iteration — plan & resolve # the concrete plan for what to try/investigate next, and why `` Fill every section from real evidence (a predicted-vs-actual number with no derivation, or a cause with no cited source, is a failed journal). This file — not memory — is how the loop compounds learning across iterations.

6c. Pipeline self-improvement (results-driven — the CHECK runs every loop, the EDIT only on real improvement). The pipeline upgrades itself, but strictly on results-ism: only a realized score improvement can trigger edits. After gap --log (and the iteration journal), run: ``bash python -m kloop.selfimprove check # pass --name when multiple projects run concurrently ``

• improved: false → touch nothing. Log the skip and move on:

python -m kloop.selfimprove log --action no_improvement --analysis "". If the previous loop's entry (kloop.selfimprove list) shows pipeline edits and this round regressed, treat them as regression suspects: restore the prior content (git diff / git checkout -- , or Edit back) and log --action reverted.

• improved: true (especially significant: true) → success retrospective first. From

the ledger, experiments/results/ and the journal, identify which bet/lever caused the delta (cite the evidence) and add a short "What worked & why" note to this iteration's journal (6b). Then ask: **is there a generalizable process lesson** — something that would help any competition, not just this one? Competition-specific tricks stay in recon.md / the journal, never in the shared pipeline.

• If a generalizable lesson exists, edit the pipeline directly — .claude/skills/**,

.claude/hooks/**, CLAUDE.md — via the Edit/Write tools (pre-authorized; no approval prompt). Read kloop.selfimprove list first so you never silently re-apply an idea a past loop reverted. Keep diffs small and surgical; preserve each SKILL.md's frontmatter. Invariants you may never weaken: the scout human gate, guard_submission's gate-before-submit enforcement, the leakage / judge-rubric gate requirements, journal append-only enforcement, autopilot bounds, "never fabricate scores".

• After editing any hook: python -m kloop.selfimprove hookcheck must pass (syntax +

smoke-run of every hook). A broken hook is worse than no improvement — restore immediately if it fails. New hooks need settings.json wiring, which is out of self-edit scope: propose that to the human via the journal instead.

• Log the outcome — mandatory every loop, whatever happened:

``bash python -m kloop.selfimprove log --action improved_and_changed \ --analysis "" --files "" \ --rationale "" --delta --gap-closed-frac python -m kloop.journal log --kind self_improve --decision "" \ --rationale "" ` (--action improvednochange` when the score improved but nothing generalizable emerged.)

1. Loop decision (journaled as loop_decision):

• Target met (python -m kloop.project gap shows target_met: true): finalize.

```bash python -m kloop.project set --stage submit --status done \

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: qurore
• Source: qurore/kaggloop
• License: MIT

Install and usage instructions live in the source repository linked above.