0002: Co-partitioning handshake for streaming merges
Status: accepted (Phase 0, ITS-112). Needs review from both tracks.
Context
A merge like orders + order_items → documents can stream partition by
partition only if every child row is read in the same unit of work as its
parent. That needs the planner (Track B) to tell the source partitioner
(Track A) where to cut. Agreeing on this late would force both sides to be
rewritten.
Decision
- The planner plans the driver entity normally (e.g.
ordersonid). - It derives
CoPartition::from_partitions("order_id", &driver): the driver’s range boundaries, applied to the child’s join column. This only works if the driver’s partitions are contiguous key ranges on one column that are unbounded at both ends; otherwise it returnsCoPartitionUnsupported. - It plans the child with
PartitionHint { co_partition: Some(..), .. }. A source must then produce exactlyCoPartition::ranges(), in order, on that column, ignoring its own sizing. A source that can’t (no such column, no efficient range scan on it) returnsCoPartitionUnsupported. align(&driver, &child)pairs partition i with partition i and refuses lists whose bounds differ. The runner leases a pair as one unit of work and feeds both streams to the merge’sTransformPlan.- On
CoPartitionUnsupported, the planner falls back to staging the smaller side and hash-joining. It shows the chosen strategy inferry plan.
Consequences
- Child partitions can be very uneven (one order with 100k items). That’s
accepted for v0.1. The child’s
target_rowsis ignored when co-partitioning. - Boundaries are
KeyValues, so driver key and child join column must compare as the same kind (both integers, both strings, both bytes). - Track A (A1, A2) implements step 3 for Postgres and Mongo. Track B (B6) implements steps 1, 2, 4 and 5.