Class HCatalogIO.Read
- All Implemented Interfaces:
Serializable
,HasDisplayData
- Enclosing class:
HCatalogIO
PTransform
to read data using HCatalog.- See Also:
-
Field Summary
Fields inherited from class org.apache.beam.sdk.transforms.PTransform
annotations, displayData, name, resourceHints
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionPCollection
<org.apache.hive.hcatalog.data.HCatRecord> Override this method to specify how thisPTransform
should be expanded on the givenInputT
.void
populateDisplayData
(DisplayData.Builder builder) Register display data for the given transform or component.withConfigProperties
(Map<String, String> configProperties) Sets the configuration properties like metastore URI.withDatabase
(String database) Sets the database name.withFilter
(String filter) Sets the filter details.withPartitionCols
(List<String> partitionCols) Set the names of the columns that are partitions.withPollingInterval
(Duration pollingInterval) If specified, polling for new partitions will happen at this periodicity.Sets the table name to read from.withTerminationCondition
(Watch.Growth.TerminationCondition<HCatalogIO.Read, ?> terminationCondition) If specified, the poll function will stop polling after the termination condition has been satisfied.Methods inherited from class org.apache.beam.sdk.transforms.PTransform
addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setDisplayData, setResourceHints, toString, validate, validate
-
Constructor Details
-
Read
public Read()
-
-
Method Details
-
withConfigProperties
Sets the configuration properties like metastore URI. -
withDatabase
Sets the database name. This is optional, assumes 'default' database if none specified -
withTable
Sets the table name to read from. -
withFilter
Sets the filter details. This is optional, assumes none if not specified -
withPollingInterval
If specified, polling for new partitions will happen at this periodicity. The returned PCollection will be unbounded. However if a withTerminationCondition is set along with pollingInterval, polling will stop after the termination condition has been met. -
withPartitionCols
Set the names of the columns that are partitions. -
withTerminationCondition
public HCatalogIO.Read withTerminationCondition(Watch.Growth.TerminationCondition<HCatalogIO.Read, ?> terminationCondition) If specified, the poll function will stop polling after the termination condition has been satisfied. -
expand
Description copied from class:PTransform
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.NOTE: This method should not be called directly. Instead apply the
PTransform
should be applied to theInputT
using theapply
method.Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
- Specified by:
expand
in classPTransform<PBegin,
PCollection<org.apache.hive.hcatalog.data.HCatRecord>>
-
populateDisplayData
Description copied from class:PTransform
Register display data for the given transform or component.populateDisplayData(DisplayData.Builder)
is invoked by Pipeline runners to collect display data viaDisplayData.from(HasDisplayData)
. Implementations may callsuper.populateDisplayData(builder)
in order to register display data in the current namespace, but should otherwise usesubcomponent.populateDisplayData(builder)
to use the namespace of the subcomponent.By default, does not register any display data. Implementors may override this method to provide their own display data.
- Specified by:
populateDisplayData
in interfaceHasDisplayData
- Overrides:
populateDisplayData
in classPTransform<PBegin,
PCollection<org.apache.hive.hcatalog.data.HCatRecord>> - Parameters:
builder
- The builder to populate with display data.- See Also:
-