public class BigQueryStorageTableSource<T> extends BoundedSource<T>
Source
representing reading from a table.BoundedSource.BoundedReader<T>
Source.Reader<T>
Modifier and Type | Field and Description |
---|---|
protected BigQueryServices |
bqServices |
protected @Nullable com.google.cloud.bigquery.storage.v1.DataFormat |
format |
protected Coder<T> |
outputCoder |
protected SerializableFunction<SchemaAndRecord,T> |
parseFn |
protected @Nullable ValueProvider<java.lang.String> |
rowRestrictionProvider |
protected @Nullable ValueProvider<java.util.List<java.lang.String>> |
selectedFieldsProvider |
Modifier and Type | Method and Description |
---|---|
static <T> BigQueryStorageTableSource<T> |
create(ValueProvider<TableReference> tableRefProvider,
com.google.cloud.bigquery.storage.v1.DataFormat format,
@Nullable ValueProvider<java.util.List<java.lang.String>> selectedFields,
@Nullable ValueProvider<java.lang.String> rowRestriction,
SerializableFunction<SchemaAndRecord,T> parseFn,
Coder<T> outputCoder,
BigQueryServices bqServices,
boolean projectionPushdownApplied) |
static <T> BigQueryStorageTableSource<T> |
create(ValueProvider<TableReference> tableRefProvider,
@Nullable ValueProvider<java.util.List<java.lang.String>> selectedFields,
@Nullable ValueProvider<java.lang.String> rowRestriction,
SerializableFunction<SchemaAndRecord,T> parseFn,
Coder<T> outputCoder,
BigQueryServices bqServices) |
BoundedSource.BoundedReader<T> |
createReader(PipelineOptions options)
Returns a new
BoundedSource.BoundedReader that reads from this source. |
long |
getEstimatedSizeBytes(PipelineOptions options)
An estimate of the total size (in bytes) of the data that would be read from this source.
|
Coder<T> |
getOutputCoder()
Returns the
Coder to use for the data read from this source. |
protected Table |
getTargetTable(BigQueryOptions options)
Returns the table to read from at split time.
|
protected java.lang.String |
getTargetTableId(BigQueryOptions options) |
void |
populateDisplayData(DisplayData.Builder builder)
Register display data for the given transform or component.
|
java.util.List<org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageStreamSource<T>> |
split(long desiredBundleSizeBytes,
PipelineOptions options)
Splits the source into bundles of approximately
desiredBundleSizeBytes . |
getDefaultOutputCoder, validate
protected final @Nullable com.google.cloud.bigquery.storage.v1.DataFormat format
protected final @Nullable ValueProvider<java.util.List<java.lang.String>> selectedFieldsProvider
protected final @Nullable ValueProvider<java.lang.String> rowRestrictionProvider
protected final SerializableFunction<SchemaAndRecord,T> parseFn
protected final Coder<T> outputCoder
protected final BigQueryServices bqServices
public static <T> BigQueryStorageTableSource<T> create(ValueProvider<TableReference> tableRefProvider, com.google.cloud.bigquery.storage.v1.DataFormat format, @Nullable ValueProvider<java.util.List<java.lang.String>> selectedFields, @Nullable ValueProvider<java.lang.String> rowRestriction, SerializableFunction<SchemaAndRecord,T> parseFn, Coder<T> outputCoder, BigQueryServices bqServices, boolean projectionPushdownApplied)
public static <T> BigQueryStorageTableSource<T> create(ValueProvider<TableReference> tableRefProvider, @Nullable ValueProvider<java.util.List<java.lang.String>> selectedFields, @Nullable ValueProvider<java.lang.String> rowRestriction, SerializableFunction<SchemaAndRecord,T> parseFn, Coder<T> outputCoder, BigQueryServices bqServices)
public void populateDisplayData(DisplayData.Builder builder)
Source
populateDisplayData(DisplayData.Builder)
is invoked by Pipeline runners to collect
display data via DisplayData.from(HasDisplayData)
. Implementations may call super.populateDisplayData(builder)
in order to register display data in the current namespace,
but should otherwise use subcomponent.populateDisplayData(builder)
to use the namespace
of the subcomponent.
By default, does not register any display data. Implementors may override this method to provide their own display data.
populateDisplayData
in interface HasDisplayData
populateDisplayData
in class Source<T>
builder
- The builder to populate with display data.HasDisplayData
public long getEstimatedSizeBytes(PipelineOptions options) throws java.lang.Exception
BoundedSource
If there is no way to estimate the size of the source implementations MAY return 0L.
getEstimatedSizeBytes
in class BoundedSource<T>
java.lang.Exception
protected java.lang.String getTargetTableId(BigQueryOptions options) throws java.lang.Exception
java.lang.Exception
protected Table getTargetTable(BigQueryOptions options) throws java.lang.Exception
java.lang.Exception
public Coder<T> getOutputCoder()
Source
Coder
to use for the data read from this source.getOutputCoder
in class Source<T>
public java.util.List<org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageStreamSource<T>> split(long desiredBundleSizeBytes, PipelineOptions options) throws java.lang.Exception
BoundedSource
desiredBundleSizeBytes
.split
in class BoundedSource<T>
java.lang.Exception
public BoundedSource.BoundedReader<T> createReader(PipelineOptions options) throws java.io.IOException
BoundedSource
BoundedSource.BoundedReader
that reads from this source.createReader
in class BoundedSource<T>
java.io.IOException