Package org.apache.beam.sdk.io.hcatalog

Class HCatalogIO.Read

java.lang.Object
org.apache.beam.sdk.transforms.PTransform<PBegin,PCollection<org.apache.hive.hcatalog.data.HCatRecord>>
org.apache.beam.sdk.io.hcatalog.HCatalogIO.Read
All Implemented Interfaces:
Serializable, HasDisplayData
Enclosing class:
HCatalogIO
public abstract static class HCatalogIO.Read extends PTransform<PBegin,PCollection<org.apache.hive.hcatalog.data.HCatRecord>>
A PTransform to read data using HCatalog.
See Also:

  • Constructor Details

    • Read

      public Read()

  • Method Details

    • withConfigProperties

      public HCatalogIO.Read withConfigProperties(Map<String,String> configProperties)
      Sets the configuration properties like metastore URI.

    • withDatabase

      public HCatalogIO.Read withDatabase(String database)
      Sets the database name. This is optional, assumes 'default' database if none specified

    • withTable

      public HCatalogIO.Read withTable(String table)
      Sets the table name to read from.

    • withFilter

      public HCatalogIO.Read withFilter(String filter)
      Sets the filter details. This is optional, assumes none if not specified

    • withPollingInterval

      public HCatalogIO.Read withPollingInterval(Duration pollingInterval)
      If specified, polling for new partitions will happen at this periodicity. The returned PCollection will be unbounded. However if a withTerminationCondition is set along with pollingInterval, polling will stop after the termination condition has been met.

    • withPartitionCols

      public HCatalogIO.Read withPartitionCols(List<String> partitionCols)
      Set the names of the columns that are partitions.

    • withTerminationCondition

      public HCatalogIO.Read withTerminationCondition(Watch.Growth.TerminationCondition<HCatalogIO.Read,?> terminationCondition)
      If specified, the poll function will stop polling after the termination condition has been satisfied.

    • expand

      public PCollection<org.apache.hive.hcatalog.data.HCatRecord> expand(PBegin input)
      Description copied from class: PTransform
      Override this method to specify how this PTransform should be expanded on the given InputT.

      NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

      Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

      Specified by:
      expand in class PTransform<PBegin,PCollection<org.apache.hive.hcatalog.data.HCatRecord>>

    • populateDisplayData

      public void populateDisplayData(DisplayData.Builder builder)
      Description copied from class: PTransform
      Register display data for the given transform or component.

      populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

      By default, does not register any display data. Implementors may override this method to provide their own display data.

      Specified by:
      populateDisplayData in interface HasDisplayData
      Overrides:
      populateDisplayData in class PTransform<PBegin,PCollection<org.apache.hive.hcatalog.data.HCatRecord>>
      Parameters:
      builder - The builder to populate with display data.
      See Also: