Package org.apache.beam.sdk.io.iceberg.cdc

Class SnapshotWindowFn

java.lang.Object

org.apache.beam.sdk.transforms.windowing.WindowFn<Object,IntervalWindow>

org.apache.beam.sdk.transforms.windowing.NonMergingWindowFn<Object,IntervalWindow>

org.apache.beam.sdk.io.iceberg.cdc.SnapshotWindowFn

All Implemented Interfaces:

Serializable, HasDisplayData

public class SnapshotWindowFn
extends NonMergingWindowFn<Object,IntervalWindow>

A WindowFn that assigns each element to a 1-millisecond IntervalWindow anchored at the element's event timestamp.

We set the element's timestamp as its snapshot commit timestamp. All tasks/records from the same snapshot land in the same window.

With the per-snapshot watermark from WatchForSnapshotsSdf, the CoGroupByKey fires when a snapshot is fully drained. The watermark advances past the snapshot's commit time only after every downstream stage has finished processing that snapshot's records.

Two snapshots committed within the same millisecond may collapse into the same window. But that's okay because ReadFromChangelogs includes snapshot sequence number in the key before routing to the CoGBK, so it won't produce incorrect joins.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.windowing.WindowFn

WindowFn.AssignContext, WindowFn.MergeContext

Constructor Summary

Constructors

Constructor	Description
SnapshotWindowFn()

Method Summary

Modifier and Type	Method	Description
Collection<IntervalWindow>	assignWindows(WindowFn<Object,IntervalWindow>.AssignContext c)	Given a timestamp and element, returns the set of windows into which it should be placed.
boolean	equals(@Nullable Object obj)
WindowMappingFn<IntervalWindow>	getDefaultWindowMappingFn()	Returns the default WindowMappingFn to use to map main input windows to side input windows.
int	hashCode()
boolean	isCompatible(WindowFn<?,?> other)	Returns whether this performs the same merging as the given WindowFn.
Coder<IntervalWindow>	windowCoder()	Returns the Coder used for serializing the windows used by this windowFn.

Methods inherited from class org.apache.beam.sdk.transforms.windowing.NonMergingWindowFn

isNonMerging, mergeWindows

Methods inherited from class org.apache.beam.sdk.transforms.windowing.WindowFn

assignsToOneWindow, getWindowTypeDescriptor, populateDisplayData, verifyCompatibility

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SnapshotWindowFn

public SnapshotWindowFn()

Method Details

assignWindows

public Collection<IntervalWindow> assignWindows(WindowFn<Object,IntervalWindow>.AssignContext c)

Description copied from class: WindowFn

Given a timestamp and element, returns the set of windows into which it should be placed.

Specified by:

assignWindows in class WindowFn<Object,IntervalWindow>

isCompatible

public boolean isCompatible(WindowFn<?,?> other)

Description copied from class: WindowFn

Returns whether this performs the same merging as the given WindowFn.

Specified by:

isCompatible in class WindowFn<Object,IntervalWindow>

windowCoder

public Coder<IntervalWindow> windowCoder()

Description copied from class: WindowFn

Returns the Coder used for serializing the windows used by this windowFn.

Specified by:

windowCoder in class WindowFn<Object,IntervalWindow>

getDefaultWindowMappingFn

public WindowMappingFn<IntervalWindow> getDefaultWindowMappingFn()

Description copied from class: WindowFn

Returns the default WindowMappingFn to use to map main input windows to side input windows. This should accept arbitrary main input windows, and produce a BoundedWindow that can be produced by this WindowFn.

Specified by:

getDefaultWindowMappingFn in class WindowFn<Object,IntervalWindow>

equals

public boolean equals(@Nullable Object obj)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object