Package org.apache.beam.sdk.io

Class FileBasedSink.DynamicDestinations<UserT,DestinationT,OutputT>

java.lang.Object
org.apache.beam.sdk.io.FileBasedSink.DynamicDestinations<UserT,DestinationT,OutputT>
All Implemented Interfaces:
Serializable, HasDisplayData
Direct Known Subclasses:
DynamicAvroDestinations
Enclosing class:
FileBasedSink<UserT,DestinationT,OutputT>
public abstract static class FileBasedSink.DynamicDestinations<UserT,DestinationT,OutputT> extends Object implements HasDisplayData, Serializable
A class that allows value-dependent writes in FileBasedSink.

Users can define a custom type to represent destinations, and provide a mapping to turn this destination type into an instance of FileBasedSink.FilenamePolicy.

See Also:

  • Constructor Details

    • DynamicDestinations

      public DynamicDestinations()

  • Method Details

    • getSideInputs

      public List<PCollectionView<?>> getSideInputs()
      Override to specify that this object needs access to one or more side inputs. This side inputs must be globally windowed, as they will be accessed from the global window.

    • sideInput

      protected final <SideInputT> SideInputT sideInput(PCollectionView<SideInputT> view)
      Returns the value of a given side input. The view must be present in getSideInputs().

    • formatRecord

      public abstract OutputT formatRecord(UserT record)
      Convert an input record type into the output type.

    • getDestination

      public abstract DestinationT getDestination(UserT element)
      Returns an object that represents at a high level the destination being written to. May not return null. A destination must have deterministic hash and equality methods defined.

    • getDefaultDestination

      public abstract DestinationT getDefaultDestination()
      Returns the default destination. This is used for collections that have no elements as the destination to write empty files to.

    • getDestinationCoder

      public @Nullable Coder<DestinationT> getDestinationCoder()
      Returns the coder for FileBasedSink.DynamicDestinations. If this is not overridden, then the coder registry will be use to find a suitable coder. This must be a deterministic coder, as FileBasedSink.DynamicDestinations will be used as a key type in a GroupByKey.

    • getFilenamePolicy

      public abstract FileBasedSink.FilenamePolicy getFilenamePolicy(DestinationT destination)
      Converts a destination into a FileBasedSink.FilenamePolicy. May not return null.

    • populateDisplayData

      public void populateDisplayData(DisplayData.Builder builder)
      Populates the display data.
      Specified by:
      populateDisplayData in interface HasDisplayData
      Parameters:
      builder - The builder to populate with display data.
      See Also: