Portolan Consume

Portolan Catalog Consumption Skill

You are helping a user query and explore data from a Portolan catalog. Portolan produces optimized cloud-native geospatial data (GeoParquet for vectors, COG for rasters) with rich STAC metadata.

Portolan Catalog Consumption Guide

Help users query and explore Portolan catalogs. Portolan produces optimized cloud-native geospatial data (GeoParquet for vectors, COG for rasters) with rich STAC metadata.

Step 1: Detect User's Environment

Before suggesting tools, check what's installed:

# Check for DuckDB
which duckdb 2>/dev/null && echo "DuckDB CLI available"
python -c "import duckdb; print(f'DuckDB Python: {duckdb.__version__}')" 2>/dev/null

# Check for Python geospatial stack
python -c "import geopandas; print(f'GeoPandas: {geopandas.__version__}')" 2>/dev/null
python -c "import rioxarray; print('rioxarray available')" 2>/dev/null
python -c "import rasterio; print(f'rasterio: {rasterio.__version__}')" 2>/dev/null

Recommendations based on environment:

| Installed | For Vectors | For Rasters | |-----------|-------------|-------------| | DuckDB | DuckDB + spatial extension (best) | N/A | | GeoPandas only | gpd.read_parquet() | N/A | | rioxarray | N/A | rioxarray.open_rasterio() (best) | | rasterio only | N/A | rasterio.open() | | Nothing | Suggest DuckDB installation | Suggest rioxarray |

If user has nothing installed, explain options and suggest cloud-native approach (DuckDB for vectors, rioxarray for rasters). Offer to guide through installation but respect if they don't want to install.

Step 2: Understand the Catalog Structure

Read STAC metadata before querying data:

# For remote catalogs
curl -s "https://data.source.coop/user/catalog/collection/collection.json" | jq '.id, .title, .description'

# Check assets
curl -s "https://data.source.coop/user/catalog/collection/collection.json" | jq '.assets | keys'

# Check schema (table:columns)
curl -s "https://data.source.coop/user/catalog/collection/collection.json" | jq '."table:columns"'

For local catalogs, read the JSON files directly.

Key STAC locations:

• catalog.json — root, lists collections
• collection.json — collection metadata, schema, vector assets live here
• item.json — item metadata, raster assets live here

Step 3: URL Protocol Handling

Convert remote storage URLs to consumption URLs:

| Storage URL | Consumption URL | Notes | |-------------|-----------------|-------| | s3://data.source.coop/... | https://data.source.coop/... | Source Coop serves HTTPS | | s3://private-bucket/... | s3://private-bucket/... | Needs credential config | | gs://bucket/... | https://storage.googleapis.com/bucket/... | For public GCS | | Local path | file:///path/to/... | Works with DuckDB |

For private S3, configure credentials:

-- DuckDB
SET s3_region = 'us-east-1';
SET s3_access_key_id = getenv('AWS_ACCESS_KEY_ID');
SET s3_secret_access_key = getenv('AWS_SECRET_ACCESS_KEY');

Step 4: Portolan GeoParquet Optimizations

Portolan produces optimized GeoParquet files. Understand these to write efficient queries:

| Optimization | Description | How to Leverage | |--------------|-------------|-----------------| | Hilbert spatial ordering | Features sorted by Hilbert curve | Spatial queries read sequentially | | Row groups (~100K rows) | Data chunked for parallel access | Predicate pushdown skips irrelevant groups | | ZSTD compression | Fast decompression, small files | Automatic, no action needed | | bbox struct column | bbox.xmin/xmax/ymin/ymax | Use for fast spatial pre-filter |

Fast spatial filtering with bbox struct:

-- Step 1: Fast filter using bbox (doesn't parse geometry)
SELECT * FROM read_parquet('https://...')
WHERE bbox.xmin > -58.5 AND bbox.xmax  -34.7 AND bbox.ymax  -58.5 AND bbox.xmax  -34.7 AND bbox.ymax  10000;

Aggregation:

SELECT
  province,
  SUM(population) as total_pop,
  COUNT(*) as num_areas
FROM read_parquet('https://...')
GROUP BY province;

Join multiple assets:

-- When collection has related tables (check STAC metadata for relationships)
SELECT r.*, c.population, c.households
FROM read_parquet('https://.../radios.parquet') r
JOIN read_parquet('https://.../census-data.parquet') c
  ON r.cod_2022 = c.id_geo;

Step 8: Partitioned Datasets

For datasets with partition:glob in STAC:

-- Query all partitions (use glob pattern from STAC)
SELECT * FROM read_parquet(
  'https://data.source.coop/user/catalog/collection/kdtree_cell=*/*.parquet'
) LIMIT 10;

-- Query single partition first (faster for exploration)
SELECT * FROM read_parquet(
  'https://data.source.coop/user/catalog/collection/kdtree_cell=0/*.parquet'
);

-- DuckDB automatically prunes partitions for Hive-style paths
SELECT * FROM read_parquet(
  'https://.../kdtree_cell=*/*.parquet'
) WHERE kdtree_cell = '42';

Troubleshooting

403 Forbidden:

• Check if bucket is public
• For private buckets, configure S3 credentials
• Source Coop uses HTTPS, not S3 protocol

Slow queries:

• Always LIMIT during exploration
• Use bbox struct for spatial pre-filtering
• Check file size in STAC (file:size) before querying large files
• For partitioned data, query one partition first

Schema mismatch:

• Read table:columns from STAC first
• Column names are case-sensitive
• Geometry column is usually geometry (check STAC)

Memory issues:

• Use DuckDB (streams data, low memory)
• For GeoPandas, read with columns= parameter to limit columns
• For rasters, use window reads

Tips

• Always read STAC metadata first — it has schema, extent, file sizes
• Always LIMIT during exploration — don't load full dataset until you know what you need
• Use bbox struct — faster than full geometry intersection
• Portolan's Hilbert ordering means spatial queries are I/O efficient
• Check for partitioning — partition:glob in STAC indicates partitioned dataset

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: portolan-sdi
• Source: portolan-sdi/portolan-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.