# Portolan Migrate > Bring an existing non-compliant catalog or published dataset into Portolan compliance without rebuilding it — audit what is there, decide whether to patch or re-extract, repair metadata, styles and data, prove conformance, then republish and prune what went stale. Use when a catalog already exists and falls short of the spec, when a dataset was published before Portolan, or when someone says a ca… - **Type:** Skill - **Install:** `agentstack add skill-portolan-sdi-portolan-skills-portolan-migrate` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [portolan-sdi](https://agentstack.voostack.com/s/portolan-sdi) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** Apache-2.0 - **Upstream author:** [portolan-sdi](https://github.com/portolan-sdi) - **Source:** https://github.com/portolan-sdi/portolan-skills/tree/main/skills/portolan-migrate ## Install ```sh agentstack add skill-portolan-sdi-portolan-skills-portolan-migrate ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Portolan Migrate Something already exists. A catalog built by an older toolchain, a folder of GeoParquet with a README, a Source Cooperative product predating Portolan. It has users, download counts, and URLs other people have written down. Your job is to bring it into compliance without breaking any of that. Use `portolan-bootstrap` when there is no catalog yet and you are converting a data source. Use this skill when the artifact exists and must survive. ## Preserve What Already Exists The data is innocent until a named requirement proves otherwise. That single rule decides most of what follows. Do not re-download upstream, reconstruct from source, filter rows, rename meaningful fields, or change CRS, geometry, or partitioning because a rewrite would be tidier. When a rule forces a change, quantify it. Report before and after row count, file count, and size, say which rule forced it, and say whether it is reversible: ``` files 235 -> 235 rows 256,555,010 -> 256,555,010 length 54,225,233 km -> 54,225,233 km row groups 376 -> 2,709 max rg rows 3,593,665 -> 100,000 (PTL-DAT-008 caps at 150,000) bytes 11,415,256,478 -> 11,999,057,592 (+5.11%) ``` That table is the deliverable for any data change. Without it you cannot tell a repair from a corruption. ## Principles **Audit before you plan.** The plan's value is showing how little work is genuinely required. One migration ran a fourteen-point data audit and found exactly one mandatory change. **Falsify your green run.** A validator that has never failed on this catalog has told you nothing. Plant a known violation and confirm it fires before you trust a clean pass. **Measure, then write.** Every derived number in the documentation comes from a query you ran over the whole dataset, not a sample and not a plausible guess. **The old catalog is live.** Its defects are being served right now, and its stale objects will survive your push. Treat the remote as something to reconcile, not something that updates itself. **Ask the publisher.** Licence, sensitivity, and exclusions are their call, not yours. Two of the three cost real rework when guessed. **Fix upstream faults upstream.** File the issue rather than papering over it here. A workaround in this catalog becomes a defect every other catalog inherits. ## Before You Touch Anything ### Back Up, and Avoid the Two Ways It Goes Wrong ```bash BK=~/backups/-premigration- rsync -a \ --exclude='.git/' --exclude='.venv/' \ --exclude='.env*' --exclude='.claude/' \ "$SRC/" "$BK/tree/" diff -r -q "$SRC" "$BK/tree" chmod -R a-w "$BK/tree" ``` **A bare `rsync -a` sweeps in credentials.** One session started copying the working tree and had to kill the run, because it was pulling in `.env.local`, `.venv`, and `.git`. Exclude explicitly, then prove the result is clean: ```bash find "$BK" \( -name '.env*' -o -name '*credential*' \ -o -name '*secret*' \) -print ``` That must return nothing. **Build the manifest in Python, not in a shell pipeline.** Terminal output decoration corrupted `find` and `ls` three separate times in one session, once producing an empty manifest that verified successfully. Walk the tree in Python, write sha256 for every file, and assert the file count you expect before declaring the backup good. Do not treat git as the backup. Data files are gitignored, so a repository is not a copy of the dataset. ### Pull the Remote in Full A diff is not enough. Fetch the remote bytes and compare them. One migration found all 178 `collection.json` files byte-identical between local and remote, which meant every defect was live. The same pull found something a metadata diff would have missed: the remote held a **curated** `.portolan/metadata.yaml` with a real title, a Spanish description, a contact, `CC-BY-4.0`, a citation, and thirteen thematic keywords. A local re-extraction had already overwritten it with `TODO: Add value`. The good metadata existed only on the server. The same pull also found 106 declared style assets that were never uploaded and returned live 404s, two corrupt 28,672-byte PMTiles that were SQLite intermediates, and stray `.pmtiles-journal` files. Against 20,513 downloads in the previous 28 days. For a large partitioned dataset, join on partition key and size rather than fetching everything, then spot-check ETags against local md5. Fall back to size alone where the ETag shows a multipart upload. Stop and ask if the two copies differ in a way that changes which is authoritative. ## Audit ### Read the Extraction Report First If the catalog came from `portolan extract`, `.portolan/extraction-report.json` names every layer that failed and why. One catalog's report read `188 layers, 179 succeeded, 9 failed`, in two clean classes: ``` Cannot do natural order without a primary key, please add ``` WFS pagination on a table with no primary key. Five layers, three attempts each. ``` Unable to merge: Field alt-piso has incompatible types: double vs int64 ``` An Arrow schema conflict between pages. Four layers. Two further failures did not appear in the report at all. One layer extracted 1,098 features whose geometries were all NULL and was dropped downstream in silence. Another failed at the tiling stage yet stayed linked in `catalog.json`, so the catalog advertised a collection with no tiles, no thumbnail, and no style. Compare the child links in `catalog.json` against the directories on disk. That catalog had 188 directories and 178 links, so ten collections existed and were unreachable. ### Count Every Defect Across Every Collection Sampling hides the shape of the problem. Count. One catalog's audit, which is a serviceable checklist for the next one: | Field | Found | Requirement | |---|---|---| | `title` | 1 of 178 | MUST | | `providers` | 0 of 178 | MUST | | Licence other than `proprietary` | 0 of 178 | `proprietary` MUST NOT be used | | `rel: describedby` | 0 of 178 | MUST | | `rel: agents` | 0 of 178 | MUST | | Portolan schema URI | 0 of 178 | MUST | | Style carrying the `default` role | 0 of 178, though 167 shipped two styles | MUST where more than one | | `AGENTS.md` on disk | 0 | MUST | | Populated temporal extent | 0 of 178 | | | `documentation` asset | 56 of 178, though 178 READMEs existed on disk | | | CRS | 63 declared EPSG:3857 over EPSG:4326 data | | Also look for unregistered custom fields, asset roles that disagree between collections, and `rel: via` links typed `text/html` where the response is GML. Then read `.portolan/metadata.yaml`. That catalog's carried literal `'TODO: Add value'` for contact, name, email, and licence, and the generated root README **published those placeholders as document text**, including a malformed `TODO: Add value `. Generated documentation propagates whatever the metadata says, so unfilled templates become published prose. Audit data from Parquet footer statistics rather than by reading rows. Nulls, minima, maxima, distinct values, geometry types, and schema drift across partitions all live in the column-chunk metadata. An entire 256-million-row integrity audit ran without touching row data. ### Classify, Then Size the Work Sort every collection into one bucket: already conformant, metadata only, documentation only, style only, thumbnail only, conversion required, data-quality problem, provenance or licence problem, or unresolved. Present the counts. That table is what tells the user whether this is an afternoon or a week. Before any conversion, run the `portolan-bootstrap` mirror-path assessment against representative upstream assets. Add its evidence and maintenance estimate to the migration checkpoint. Use a metadata-only mirror only when upstream data meets all applicable specification requirements. Otherwise, build a full mirror with conformant data copies. ## Check Whether the Toolchain Moved Before You Decide to Patch This is the highest-leverage step in the skill, and it is easy to skip. One catalog was built on 2026-06-09 with `portolan 1.0.0a0` and `gpio 1.2.0`, and nine layers had failed extraction. The plan was to patch them one at a time. Re-running extraction on the current toolchain recovered **eight of eight in 103 seconds**, where the original harvest had spent 821 seconds and failed. The reason was a single upstream commit adding a `sortBy` parameter for stable pagination on tables with no primary key, plus schema unification for the `decimal128` against `int64` conflicts. Inspecting the new output changed the plan entirely. The current CLI wrote a far better collection than the one on disk: schema URI, checksums, `AGENTS.md`, `rel: via`, `table:columns`, a `default`-roled style, and a legend harvested from WMS `GetLegendGraphic`. The new `default.json` was the publisher's own SLD cartography rather than flat blue. So before choosing: 1. Read the toolchain versions the old catalog recorded in `.portolan/`. 2. Compare against what is installed now. 3. Re-extract one representative collection and diff the result against its current form. Re-extraction can be cheaper than patching and strictly better in output. It can also be wrong, when the upstream service has changed or gone away. Make it a checkpoint and show the diff. ### Checkpoint: Patch or Re-extract Present the version delta, the sample diff, an estimated runtime for a full re-extraction, and anything the old catalog holds that a re-extraction would destroy. Ask which path to take. ## Rashid Is the Gate, and It Has a Hole You Must Plug `portolan check` runs rashid and reports `PTL-*` rule ids citing the spec requirements they enforce. Where the spec, the CLI, `stac-check`, or your own reading disagrees with rashid, rashid decides. Do not weaken it, suppress findings, or add an allow-list entry to obtain a clean run. ### Check the Version The catalog template's conformance gate resolves the validator with `shutil.which("rashid")`, so it uses whatever is on `PATH`. The gate fails when rashid is absent and when the version falls outside the range it requires, and the failure names the install command. Two ways the version still bites: * **A stale rashid under-reports.** Both migrations hit 0.1.4, which lacks `PTL-LNK-007`, `PTL-LNK-008`, `PTL-LNK-009`, and `PTL-AST-006`. The CLI's floor is 0.1.5 because 0.1.4's missing `PTL-AST-006` let a wrong COG media type mask `PTL-COL-004` and `PTL-MIR-001`. * **A rashid below 0.1.8 rejects a conforming v0.2.0 catalog.** Spec v0.2.0 retired `PORTO-CORE-034`, and rashid 0.1.8 dropped the two rules that carried it, `PTL-LNK-004` and `PTL-LNK-005`. Releases 0.1.5 through 0.1.7 still report an error for a `self` link and for an absolute structural `href`. Install `rashid>=0.1.8, The https-only rule for absolute asset hrefs does not extend to the glob: globs are consumed by partition-aware readers rather than browsers, and bucket-native schemes (`s3://`, `gs://`) MAY be used where those readers need them (glob expansion requires listing, which plain https does not provide). Enabling asterisks in HTTP paths does not rescue it. An https glob cannot work at all. Note the single-file case is unaffected, so state plainly in the documentation which access path needs credentials and which does not. **A bucket name containing dots needs path-style addressing.** Virtual-host addressing fails TLS verification. The documented setup has to say so, along with the endpoint, or the reader's first query fails. **Link checking by HTTP status does not work on Source Cooperative.** It is a single-page app and returns 200 for every path, including paths outside the product. Fetch the rendered page and read the emitted HTML. **Relative markdown links resolve one directory too high**, because the rendered root page has no trailing slash, so a bare `href="AGENTS.md"` points outside the product. Publish absolute URLs. ### Run Every Documented Query Against Published Data A gate, not a courtesy. Use only the setup the documentation gives the reader, from a clean environment. This is what caught the broken `partition:glob`, after the catalog had already been declared finished. If a documented query fails, fix the query or delete it. Do not ship it broken. Then re-run the validator against the live catalog, which probes the host for range support and CORS headers. `portolan check --live` takes the published base URL as `--url`, overriding `publish.public_url`. Running rashid directly, the same argument is spelled `--live-base-url`. ## Mechanics Read `portolan-cli` for the full command reference and `git-backed-catalog` for repository and publication mechanics. This section covers only what migration adds. Both migrations ran the repository path: a catalog repo created from `portolan-sdi/portolan-catalog-template`, generators under `tools/`, gates under `tests/`, and publication through `tools/publish.py`. Follow `git-backed-catalog` for that. Two cautions specific to migration: **A turn interrupt kills a foreground background job.** One extraction died at 84 of 187 layers while reporting exit 0. Detach long runs and poll a log for a completion sentinel: ```bash setsid nohup ./run-extract.sh > /dev/null 2>&1 < /dev/null & disown until grep -q 'DONE exit=' extract.log; do sleep 20; done ``` **Exit code 0 can mean "aborted at a confirmation prompt."** Pipe `yes |` into anything that prompts, or you will read a successful exit from a command that did nothing. **Deferring thumbnails leaves a rule firing.** Migration often stages thumbnails separately, through `portolan-thumbnails`, so `portolan add --no-thumbnails` is the common call. That leaves `PTL-VIZ-001` failing until the images land. Expect it in the interim baseline rather than chasing it, and pair `--no-thumbnails` with `check.disabled` in `.portolan/config.yaml` only for a catalog that will never ship thumbnails at all. ### Generated Metadata Overwrites Authored Metadata `portolan add` regenerates `collection.json` through hierarchical metadata resolution. In one migration that clobbered every collection's `title` with the catalog root's title, and reverted the `stac_extensions` schema URI. Assume nothing survives. `license` and `providers` are overwritten outright whenever the merged metadata carries them, and `init` seeds both at the root, so the root's values reach every collection. `description` survives only where the merged metadata is blank, and `id` survives only because the existing file is reloaded first. The `SMART` merge strategy applies to assets and items, not to collection identity fields. The schema URI is not hardcoded either. The CLI stamps the highest version bundled by the rashid wheel it has installed, so the version you get tracks a dependency rather than the spec release you are targeting. A CLI running rashid 0.1.8 stamps v0.2.0. Keep authored metadata in a generator and re-apply it as an idempotent last pass after **every** `add`. See `reference/tools/apply_metadata.py`. Add a CI gate that regenerates and diffs, so a hand-edit to generated output fails the build rather than surviving until the next regeneration wipes it. ### Reference Implementations `reference/tools/` carries six scripts lifted from the two migrations. Each names in its docstring the defect that forced it to exist. | Script | Works around | |---|---| | `reencode.py` | Row groups over the `PTL-DAT-008` cap, streamed so peak memory is one batch | | `build_collection.py` | Measured extents, counts, and checksums, with a staleness check | | `validate_with_data.py` | The data pass having nothing local to read | | `sld_graduated.py` | The SLD converter rejecting class-break styles | | `fix_styles.py` | Generated styles carrying no source URL and no zoom range | | `apply_metadata.py` | `add` overwriting authored metadata | ## Checkpoints | Checkpoint | Present | Ask | |---|---|---| | Starting state | Local and remote inventory, what differs, which is authoritative | Proceed, or resolve the discrepancy first? | … ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [portolan-sdi](https://github.com/portolan-sdi) - **Source:** [portolan-sdi/portolan-skills](https://github.com/portolan-sdi/portolan-skills) - **License:** Apache-2.0 Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** yes - **Shell / process execution:** no - **Environment & secrets:** yes - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-portolan-sdi-portolan-skills-portolan-migrate - Seller: https://agentstack.voostack.com/s/portolan-sdi - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.