-
Notifications
You must be signed in to change notification settings - Fork 193
Open
Description
The half-join operator consumes a stream of updates and joins it with an arrangement. It needs updates in the form of vectors of data, currently it cannot handle any other format.
However, it should support containers on its input to avoid forcing data into an owned representation. This involves:
- The data needs to be sorted, and consolidated.
- We could arrange the input data, and drop the trace to form batches of ready proposals.
- We could work on chunks of sorted and consolidated data. This amortizes the work of traversing the lookup arrangement, at the expense of not guaranteeing that the inputs are consolidated.
- We could use a merge batcher (without an arrangement) to force the stream inputs to be sorted and consolidated.
Since #619, we can push outputs into a container builder, which allows arbitrary containers (and transformations).
Metadata
Metadata
Assignees
Labels
No labels