Apache Hudi Lakehouse

Apache Hudi Lakehouse

Overview

Use this skill when Apache Hudi is the primary table layer for incremental lakehouse workloads. It helps agents reason about mutation-heavy patterns, table type selection, compaction behavior, timeline safety, and consumer expectations across read-optimized and real-time query paths.

When to Use

• choosing or operating Apache Hudi for lakehouse tables
• building record-level upsert or delete pipelines
• managing compaction, clustering, and incremental consumption
• supporting lakehouse tables with heavy mutations (CDC sinks, slowly changing dimensions)
• planning multi-engine access (Spark, Presto, Trino, Athena, Hive)

Do not use this when the workload is append-only with no mutation requirements and simpler formats like Parquet or Iceberg would suffice.

Workflow

1. Define mutation patterns and read access expectations.

Include:

• primary record key and partition path
• expected operations: inserts, upserts, deletes, or bulk replaces
• read latency expectations: are readers okay with merge-on-read or do they need read-optimized snapshots?
• query engines that must access the table
• expected write throughput and record mutation rate

1. Choose the right table type and indexing strategy.

• Copy-on-Write (COW): best for read-heavy workloads, produces columnar snapshots on write
• Merge-on-Read (MOR): best for write-heavy workloads, defers merge to read time or compaction
• choose record index type: BLOOM, GLOBAL_BLOOM, SIMPLE, BUCKET, or RECORD_INDEX
• index choice affects upsert performance and scaling behavior
• document why the table type was chosen — revisiting later is expensive

1. Plan compaction and clustering explicitly.

• for MOR tables: compaction converts log files to columnar — it is not optional
• define compaction strategy: synchronous (inline) or asynchronous (scheduled)
• set compaction triggers: by number of commits, time, or log file size
• clustering reorganizes data layout for query performance — plan separately from compaction
• budget compute for compaction and clustering in cost planning

1. Design incremental consumption and downstream contracts.

• Hudi supports incremental queries by commit timeline
• define the consumer contract: which commit instant do consumers start from?
• plan for consumer resets and bootstrap reads
• document how schema changes affect incremental consumers
• test that consumers handle compaction and rollback instants correctly

1. Handle schema evolution and timeline safety.

• Hudi supports schema evolution but not all changes are safe across readers
• column adds are generally safe; renames and type changes require care
• define compatibility expectations per reader engine
• rollback instants can confuse consumers — document rollback behavior
• archive policy affects timeline visibility for late consumers

1. Plan operations, monitoring, and recovery.

• monitor timeline growth, pending compactions, and inflight commits
• alert on compaction backlog and write failures
• plan for rollback: Hudi supports instant-level rollback, but consumers must handle gaps
• define retention and archival for the Hudi timeline
• document backup and restore procedures for critical tables

Common Rationalizations

| Rationalization | Reality | | --- | --- | | "Hudi handles upserts so we don't need to think about keys." | Record key and partition path design determines correctness, performance, and scaling. Wrong keys cause silent data loss or duplication. | | "MOR is always better because writes are faster." | MOR defers work to compaction and read time. Without compaction planning, read performance degrades unboundedly. | | "Compaction will just happen in the background." | Compaction requires explicit scheduling, compute budget, and monitoring. Unmanaged compaction leads to reader degradation and timeline bloat. | | "All query engines see the same data." | COW and MOR tables expose different snapshots to different query types. Read-optimized queries on MOR tables see only compacted data. |

Red Flags

• record key chosen without understanding uniqueness guarantees
• MOR table with no compaction schedule or monitoring
• incremental consumers have no documented starting instant or reset procedure
• schema changes deployed without testing across all reader engines
• no monitoring of timeline growth, pending compactions, or inflight commits
• clustering is never run despite increasing query scan ranges
• rollback behavior is undocumented and consumers assume a linear timeline
• index type is default without analysis of key cardinality and write patterns

Verification

• [ ] Record key, partition path, and mutation semantics are explicitly documented
• [ ] Table type choice (COW vs MOR) is justified with read/write trade-off analysis
• [ ] Compaction is scheduled, monitored, and budgeted for compute cost
• [ ] Incremental consumer contracts define starting instants and reset behavior
• [ ] Schema evolution paths are tested across all target query engines
• [ ] Timeline monitoring covers pending compactions, inflight commits, and archival
• [ ] Rollback behavior is documented and consumers handle timeline gaps safely

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Kilo-Org
• Source: Kilo-Org/kilo-marketplace
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.