Class SparkDatasetTranslationContext
java.lang.Object
org.apache.beam.runners.spark.translation.SparkTranslationContext
org.apache.beam.runners.spark.translation.SparkDatasetTranslationContext
Translation context of the Dataset-based portable backend. Keeps one
Dataset per
translated PCollection and evaluates the ones no transform consumed in computeOutputs().-
Constructor Summary
ConstructorsConstructorDescriptionSparkDatasetTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions options, JobInfo jobInfo) -
Method Summary
Modifier and TypeMethodDescriptionvoidEvaluates every Dataset no transform consumed.<T> org.apache.spark.sql.Dataset<WindowedValue<T>> getDataset(String pCollectionId) Returns the Dataset of a PCollection and marks it as consumed.org.apache.spark.sql.SparkSessionorg.apache.spark.storage.StorageLevel<T> voidputDataset(String pCollectionId, org.apache.spark.sql.Dataset<WindowedValue<T>> dataset) Registers the Dataset of a PCollection.<T> voidputDataset(String pCollectionId, org.apache.spark.sql.Dataset<WindowedValue<T>> dataset, boolean cache) Registers the Dataset of a PCollection.<T> org.apache.spark.sql.Encoder<WindowedValue<T>> windowedEncoder(String pCollectionId, org.apache.beam.model.pipeline.v1.RunnerApi.Components components) Encoder of the windowed values of a PCollection, derived from its wire coder.Methods inherited from class org.apache.beam.runners.spark.translation.SparkTranslationContext
getSerializableOptions, getSparkContext, nextSinkId, popDataset, pushDataset
-
Constructor Details
-
SparkDatasetTranslationContext
public SparkDatasetTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions options, JobInfo jobInfo)
-
-
Method Details
-
getSparkSession
public org.apache.spark.sql.SparkSession getSparkSession() -
getStorageLevel
public org.apache.spark.storage.StorageLevel getStorageLevel() -
putDataset
public <T> void putDataset(String pCollectionId, org.apache.spark.sql.Dataset<WindowedValue<T>> dataset) Registers the Dataset of a PCollection. Datasets read by several transforms are persisted. -
putDataset
public <T> void putDataset(String pCollectionId, org.apache.spark.sql.Dataset<WindowedValue<T>> dataset, boolean cache) Registers the Dataset of a PCollection. Passcacheas false for a Dataset that is a projection of an already persisted one, so the same rows are not cached twice. -
getDataset
Returns the Dataset of a PCollection and marks it as consumed. -
windowedEncoder
public <T> org.apache.spark.sql.Encoder<WindowedValue<T>> windowedEncoder(String pCollectionId, org.apache.beam.model.pipeline.v1.RunnerApi.Components components) Encoder of the windowed values of a PCollection, derived from its wire coder. -
computeOutputs
public void computeOutputs()Evaluates every Dataset no transform consumed.- Overrides:
computeOutputsin classSparkTranslationContext
-