Lecture To Notes

Lecture-to-Notes

Turn a lecture recording (video or audio-only) into structured notes. Every heavy stage runs locally at 0 Claude tokens; Claude only does the final synthesis. This page is the map; detail lives in reference/, one topic per file.

| File | What is in it | |---|---| | reference/pipeline.md | Per-stage flags, thresholds, JSON schemas, timeouts, observability | | reference/note-spec.md | Note quality spec, tier scoring, width table, synthesis prompt requirements | | reference/segmented-mode.md | Multi-talk workshop folders → per-segment L2/L3 + Hub + web viewer | | reference/multi-camera.md | One long recording + many phone clips/photos → one timeline | | reference/decisions.md | Post-mortems, benchmarks, wrong turns, VRAM measurements |

HARD RULES

1. ==ASK the user what language the speaker(s) used== (English / Mandarin /

bilingual code-switching) before transcribing. There is no default and transcribe_video.py exits without --lang. A wrong guess makes Whisper hallucinate Chinese from accented English and the transcript is unusable.

1. ==Never skip Stage D (VLM) or Stage E (grounding)== for speed or for a

deadline. ==The user has not set a deadline; do not invent one.== If a stage really is too slow (>2 h ETA), report the ETA and ask.

1. ==Never auto-correct the transcript.== Flag suspects, let synthesis resolve

them. Both auto-correction passes ever built were measured and retired — see reference/decisions.md#asr-auto-correction. Do not add an auto-apply mode.

1. ==Do not bypass the collapse auto-retry.== transcribe_video.py auto-runs

retranscribe_segment.py --auto on detected token-collapse. If collapses survive that, escalate (wider beam + --no-repeat-ngram-size + a glossary), never skip.

1. ==The VLM does not do OCR.== Stage D asks only for semantic signals. Text

comes from Stage B quick_text, Stage B2 clean_text, or pdf_text.json. ocr.vlm_text is an always-empty compatibility field.

1. ==Serialize all GPU work.== Whisper and the VLM may not run concurrently on an

8 GB card, and frame extraction must not run alongside transcription.

1. ==On an 8 GB card, never exceed --batch-size 4 with --beam-size 10==, and

never combine --beam-size 15 with sequential mode — that crashes (0xC0000005). The measured sweet spot is --batch-size 3 --beam-size 10.

1. ==Director batch dispatch==: for a multi-lecture batch, dispatch Steps 1–9 as

one subagent and Step 10 synthesis as a separate fresh subagent, spawned only after slides_grounded.json exists. A single subagent bounces during the long GPU waits and burns 30+ min of wall time per lecture.

1. ==🚫 PHI red line==: if the recording contains patient-identifiable content

(case discussion, ward rounds, named patients), transcribe LOCAL ONLY — drop --engine groq. When unsure, ask; default to local.

Input types and routing

==Start here. route_inputs.py is the front door== — it classifies a folder and prints the ordered commands plus the questions a human must answer. It is plan-only: it never runs anything and never writes a file.

python /scripts/route_inputs.py  [--recursive] [--out-dir DIR] [--json]

| What is in the folder | Slide source | Route | |---|---|---| | Video, no deck | frames from the video | Path A — Steps 5–7 | | Audio/video + PDF deck (==preferred==) | PDF text + page renders | Path B — build_slides_from_pdf.py | | Audio/video + loose slide images (≥3) | the images themselves | Path B-images — build_slides_from_images.py | | Audio only, no deck | none | Path C — transcript-only note | | N-up handout PDF | cropped tiles | Path B-multi — crop_multiup_pdf.py first | | Multi-talk workshop folder | per segment | reference/segmented-mode.md | | One long recording + many phone clips/photos | per source | reference/multi-camera.md | | .pptx / .docx / .key | — | convert to PDF yourself first; there is no conversion step here |

==Multi-source contract==: when two or more independent sources are present, run media_capture_index.py --emit-alignment alignment.json first. ==Capture timestamps are HYPOTHESES; transcript cross-correlation (xcorr_media_offsets.py) is EVIDENCE.== A source whose reliable flag is false got its start from mtime or has none — it must not be aligned on. Nothing is ever auto-corrected: a claimed-vs-measured disagreement >5 s is flagged "conflict": true for a human to judge. Details in reference/multi-camera.md.

Pipeline

One command plus its purpose per step; flags, thresholds and outputs are in reference/pipeline.md.

Step 1 — Ask the language (mandatory, no command)

English / Mandarin / bilingual? Accented speakers? Code-switching mid-sentence? Use AskUserQuestion if the user has not said. HARD RULE 1.

Step 2 — Set up the lecture directory

One directory per lecture holds every intermediate; name it {date}_{speaker}_{topic}, the shape finalize_to_vault.py parses.

Step 3 — GPU pre-flight

python /scripts/gpu_check.py --out-dir "$OUT_DIR" --min-free-mb 6000

Gate before transcription and again before Stage D. Exit 0 proceed, 1 warn and proceed, 2 blocked — surface it, ==do not retry in a loop==. → reference/pipeline.md#gpu-check

Step 4 — Transcribe

python /scripts/transcribe_video.py "" \
    --output-dir "$OUT_DIR" --lang  \
    --batch-size 3 --beam-size 10

Local faster-whisper by default; --engine groq is an optional offload (HARD RULE 9). Default model alias is breeze25 (needs a local model dir); on a machine without one, pass --model large-v3, which faster-whisper downloads. Recordings over ~30 min go through the chunked runner instead. → reference/pipeline.md#transcription

Step 5 — Stage A: frame extraction (Path A only)

python /scripts/extract_slides.py "" --output-dir "$OUT_DIR" --interval 15

Writes slides/frame_NNNN.jpg + slides/timestamps.json, phash-deduping adjacent near-identical frames. Path B/B-images skip this. → reference/pipeline.md#stage-a

Step 6 — Stage B: quick OCR + entropy (Path A only)

python /scripts/quick_ocr.py "$OUT_DIR"

RapidOCR on every frame → slides_raw.json. ==Required==: without it every slide looks decorative to the Stage D gate. → reference/pipeline.md#stage-b

Step 6-alt — Path B / B-images bridge

python /scripts/build_slides_from_pdf.py    "$OUT_DIR"   [--audio-duration-sec N]
python /scripts/build_slides_from_images.py "" -o "$OUT_DIR" [--audio-duration-sec N]

Either bridge emits slides_raw.json + slides_dedup.json directly, replacing Steps 5–7. → reference/pipeline.md#path-b

Step 7 — Stage C: semantic dedup (Path A only)

python /scripts/dedup_semantic.py "$OUT_DIR"

Merges adjacent frames by text-subset or layout similarity, marks dedup.is_canonical. Output slides_dedup.json. → reference/pipeline.md#stage-c

Step 8 — Stage B2: high-quality OCR (Surya)

python /scripts/ocr_surya.py "$OUT_DIR" [--resume]

Surya in its own venv on canonical text-bearing slides, RapidOCR as the shallow fallback. Adds ocr.clean_text / ocr_engine / ocr_confidence. ==Updates slides_dedup.json in place== (one-time backup slides_dedup.pre_b2.json) and writes slides_ocr.json. Path B skips it — pdf_text is already clean. Without a Surya venv it warns and routes everything to RapidOCR rather than failing. → reference/pipeline.md#stage-b2

Step 9 — Stage D: VLM signals

python /scripts/vlm_signals.py "$OUT_DIR" --model minicpm-v:8b --num-ctx 4096

Semantic signals per canonical slide, behind a 4-condition pre-skip gate for decorative frames. Re-check the GPU first (Step 3). Output slides_vlm.json. scripts/ocr_slides.py is a deprecated shim forwarding here, same argv and outputs. → reference/pipeline.md#stage-d

Step 10 — Stage E: transcript grounding

python /scripts/ground_slides.py "$OUT_DIR"

Pure Python, 0 LLM calls. Ties each canonical slide to the words spoken over it. Output slides_grounded.json — the input to synthesis. → reference/pipeline.md#stage-e

Step 11 — Flag suspect ASR tokens

python /scripts/flag_asr_suspects.py --dir "$OUT_DIR"

Runs HERE, after Stage E: the slide glossary it needs comes from slides_grounded.json. Writes asr_suspects.txt; ==the transcript is left byte-identical==. Treat each line as a question, never a substitution. → reference/pipeline.md#asr-suspects

Step 12 — Chunked pre-summarization (long lectures only)

Over ~30 min / 25 k tokens of transcript, offload chunk summaries to a Sonnet subagent instead of reading the whole transcript into main context. Coverage guards ([CHUNK_END], [CONTINUE_NEEDED], expected-chunk count) are mandatory. → reference/pipeline.md#chunked-summarization

Step 13 — Stage F: synthesis (Claude)

Two passes for batches and long lectures — ==Tier-pass then Write-pass==:

• Tier-pass subagent reads slides_grounded.json + transcript.txt +

pdf_text.json, applies the tier scoring rules, writes only slides_final.json (integer tier, attachment_name, embed_width, section_suggestion). This file is the frozen tier authority.

• Write-pass subagent reads the frozen slides_final.json + transcript +

slide text, writes note_draft.md with [[EMBED sN]] placeholders only — no paths, widths or callouts.

One pass is fine for one short lecture; splitting them stops the writer from simplifying structure to make its own embed audit pass. → reference/note-spec.md (mandatory: quality spec, tier rules, prompt requirements)

Step 14 — Render, finalize, audit

python /scripts/render_embeds.py    "$OUT_DIR" --note note_draft.md --in-place
python /scripts/finalize_to_vault.py "$OUT_DIR" [--vault-root PATH]
python /scripts/audit_note.py "" --mode lecture --grounding "$OUT_DIR"

render_embeds.py expands placeholders to col-0 callouts with path + width and audits Tier-1/2 coverage; finalize_to_vault.py copies cited slides + the note into the vault; the auditor is the gate. ==Always pass --grounding== — without it the caption↔frame check only warns. → reference/note-spec.md Draft review exemption: this output is machine-transcribed and synthesized — write to the inbox without showing a draft; the user reviews in Obsidian.

Edge cases

• Audio only, no deck → transcript-only note using # 逐段筆記 instead of

# 逐投影片筆記.

• N-up handout PDF → render one mid page and ==look at it== before deciding

the grid; heuristics are unreliable on slide-heavy PDFs. Then crop_multiup_pdf.py --expected-rows R --expected-cols C. Pages that are genuinely 1-up (title pages) are handled per page, not forced into the consensus grid.

• Very long lecture (>90 min) → chunked runner for transcription, Step 12 for

reading it. Batch of recordings → transcribe strictly sequentially; ~4.5 GB RAM per faster-whisper instance.

• One talk split across several files → one note, not several.
• Slides English, speaker Mandarin → keep both; the deck gives terms, the

transcript gives the explanation.

• Speaker asked not to be recorded → exclude that content.
• Dense text handout, not a slide deck → primary source, but drop the

slide-by-slide structure.

• CJK path failures (exit 127 / 3221226505) → extract audio with ffmpeg

separately first; the script reuses a validated audio.wav.

Optional infrastructure

Everything below is ==this machine's setup, not a requirement== — nothing in the pipeline depends on any of it, and the generic alternative is inline.

| Used for | Generic alternative | |---|---| | job_runner.py wrapping long GPU jobs (tree-kills children on timeout) | plain timeout , or run in the foreground | | gpu_lease.py / a pause-flag file between concurrent batches | run GPU stages one at a time; leave paths.pause_flag empty in config | | vault-search / OpenEvidence / Zotero lookups during synthesis | skip; cite only what the lecture itself provided | | ntfy completion pings | skip | | external batch control plane (run_queue, rerun_batch, clip_order, dashboard) | course-specific, not shipped — see reference/segmented-mode.md |

Vault paths (99Attachment/lecture_{slug}, the inbox folder) are ==a private vault convention== and configurable: render_embeds.py --attach-root / --attach-dir, finalize_to_vault.py --vault-root.

Dependencies

ollama pull minicpm-v:8b               # ~5.5 GB Q4_K_M, Stage D
pip install rapidfuzz rapidocr-onnxruntime scikit-image pyyaml json_repair pillow numpy

ffmpeg + ffprobe on PATH. Surya (Stage B2) lives in its own venv; point ocr_engine.surya_python at it in config.yaml, or leave it blank to fall back to RapidOCR. Copy config.example.yaml → config.yaml on a new machine; every machine-specific value there is blank by default and env-overridable. ==Optional dependencies degrade loudly, not silently== — a missing scikit-image or RapidOCR is reported and gated, because a silent degrade produced wrong output rather than less output (reference/decisions.md#optional-dependency-degradation).

File locations

/
├── SKILL.md, config.yaml, config.example.yaml
├── reference/  pipeline.md · note-spec.md · segmented-mode.md · multi-camera.md · decisions.md
├── data/       real_words.txt, real_acronyms.txt   (regenerated, not committed)
├── ocr_bench/  engine benchmark harness (bring your own fixtures)
└── scripts/
    ├── route_inputs.py           front door — classifies material, prints the plan
    ├── transcribe_video.py  retranscribe_segment.py  gpu_check.py  groq_asr.py
    ├── extract_slides.py  quick_ocr.py  dedup_semantic.py  ocr_surya.py
    ├── vlm_signals.py            (ocr_slides.py = deprecated shim → here)
    ├── build_slides_from_pdf.py  build_slides_from_images.py  crop_multiup_pdf.py
    ├── ground_slides.py  flag_asr_suspects.py  make_glossary.py  build_real_words.py
    ├── render_embeds.py  finalize_to_vault.py  audit_note.py  export_web.py
    ├── media_capture_index.py  xcorr_media_offsets.py  query_near_field.py
    ├── adapters/    surya_adapter.py        (production OCR adapters)
    ├── batch/       build_L1 · split_segments · split_L1_by_segment · add_dhash ·
    │                vlm_cache · detect_language · detect_language_audio ·
    │                phi_mask · process_slide_deck     (generic batch layer)
    ├── layout2/     viewer.css, viewer.js   (web viewer assets, edited verbatim)
    └── _common.py  _log.py  _paths.py

Per-lecture output directory:

{lecture}/
├── metadata.json          run_id + media fingerprint + per-stage status
├── transcript.json/.txt   timestamped segments; .txt is [MM:SS] text, H:MM:SS past an hour
├── asr_suspects.txt       flagged tokens — flags only, never a rewrite
├── alignment.json         multi-source capture-start hypotheses (when applicable)
├── slides/                frame_NNNN.jpg | page_NN.jpg | original photo names
├── slides_raw.json        Stage B    quick text + density + entropy
├── slides_dedup.json      Stage C    canonical markers (Stage B2 updates in place)
├── slides_dedup.pre_b2.json          one-time pre-Stage-B2 snapshot
├── slides_ocr.json        Stage B2   Surya result, for inspection
├── slides_vlm.json        Stage D    VLM signals + vlm_skip + skip_metrics
├── slides_grounded.json   Stage E    transcript grounding + retrieval fields
├── slides_final.json      Stage F    tier + score + attachment_name + width
└── logs/progress_*.jsonl  per-stage event streams (not every stage emits one)

Each stage output is a superset of the previous, so you can re-run one stage without redoing transcription or frame extraction. runs.jsonl one level up carries one summary line per stage run, joined by run_id. Keep the intermediates — they are how tier decisions get debug

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: drpwchen
• Source: drpwchen/lecture-to-notes
• License: MIT

Install and usage instructions live in the source repository linked above.