Package org.apache.beam.sdk.io.gcp.bigquery

Class UpdateSchemaDestination<DestinationT>

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<Iterable<KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>,Iterable<KV<TableDestination,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>>

org.apache.beam.sdk.io.gcp.bigquery.UpdateSchemaDestination<DestinationT>

All Implemented Interfaces:

Serializable, HasDisplayData

public class UpdateSchemaDestination<DestinationT>
extends DoFn<Iterable<KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>,Iterable<KV<TableDestination,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>>

Update destination schema based on data that is about to be copied into it.

Unlike load and query jobs, BigQuery copy jobs do not support schema field addition or relaxation on the destination table. This DoFn fills that gap by updating the destination table schemas to be compatible with the data coming from the source table so that schemaUpdateOptions are respected regardless of whether data is loaded directly to the destination table or loaded into temporary tables before being copied into the destination.

This transform takes as input a list of KV(destination, WriteTables.Result) and emits a list of KV(TableDestination, WriteTables.Result) where the destination label is parsed and replaced to TableDestination objects.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext

Constructor Summary

Constructors

Constructor	Description
UpdateSchemaDestination(BigQueryServices bqServices, PCollectionView<String> zeroLoadJobIdPrefixView, ValueProvider<String> loadJobProjectId, BigQueryIO.Write.WriteDisposition writeDisposition, BigQueryIO.Write.CreateDisposition createDisposition, int maxRetryJobs, String kmsKey, Set<BigQueryIO.Write.SchemaUpdateOption> schemaUpdateOptions, DynamicDestinations<?,DestinationT> dynamicDestinations)

Method Summary

Modifier and Type	Method	Description
void	finishBundle(DoFn<Iterable<KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>,Iterable<KV<TableDestination,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>>.FinishBundleContext context)
void	onTeardown()
void	processElement(Iterable<KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>> element, DoFn<Iterable<KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>,Iterable<KV<TableDestination,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>>.ProcessContext context, BoundedWindow window)
void	startBundle(DoFn<Iterable<KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>,Iterable<KV<TableDestination,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>>.StartBundleContext c)

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

UpdateSchemaDestination

public UpdateSchemaDestination(BigQueryServices bqServices,
 PCollectionView<String> zeroLoadJobIdPrefixView,
 @Nullable
 ValueProvider<String> loadJobProjectId,
 BigQueryIO.Write.WriteDisposition writeDisposition,
 BigQueryIO.Write.CreateDisposition createDisposition,
 int maxRetryJobs,
 @Nullable
 String kmsKey,
 Set<BigQueryIO.Write.SchemaUpdateOption> schemaUpdateOptions,
 DynamicDestinations<?,DestinationT> dynamicDestinations)

Method Details

startBundle

@StartBundle
public void startBundle(DoFn<Iterable<KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>,Iterable<KV<TableDestination,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>>.StartBundleContext c)

processElement

@ProcessElement
public void processElement(@Element
 Iterable<KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>> element,
 DoFn<Iterable<KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>,Iterable<KV<TableDestination,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>>.ProcessContext context,
 BoundedWindow window)
                    throws IOException

Throws:

IOException

onTeardown

@Teardown
public void onTeardown()

finishBundle

@FinishBundle
public void finishBundle(DoFn<Iterable<KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>,Iterable<KV<TableDestination,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>>.FinishBundleContext context)
                  throws Exception

Throws:

Exception