Class SparkDatasetTranslationContext

java.lang.Object
org.apache.beam.runners.spark.translation.SparkTranslationContext
org.apache.beam.runners.spark.translation.SparkDatasetTranslationContext

public class SparkDatasetTranslationContext extends SparkTranslationContext
Translation context of the Dataset-based portable backend. Keeps one Dataset per translated PCollection and evaluates the ones no transform consumed in computeOutputs().
  • Constructor Details

    • SparkDatasetTranslationContext

      public SparkDatasetTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions options, JobInfo jobInfo)
  • Method Details

    • getSparkSession

      public org.apache.spark.sql.SparkSession getSparkSession()
    • getStorageLevel

      public org.apache.spark.storage.StorageLevel getStorageLevel()
    • putDataset

      public <T> void putDataset(String pCollectionId, org.apache.spark.sql.Dataset<WindowedValue<T>> dataset)
      Registers the Dataset of a PCollection. Datasets read by several transforms are persisted.
    • putDataset

      public <T> void putDataset(String pCollectionId, org.apache.spark.sql.Dataset<WindowedValue<T>> dataset, boolean cache)
      Registers the Dataset of a PCollection. Pass cache as false for a Dataset that is a projection of an already persisted one, so the same rows are not cached twice.
    • getDataset

      public <T> org.apache.spark.sql.Dataset<WindowedValue<T>> getDataset(String pCollectionId)
      Returns the Dataset of a PCollection and marks it as consumed.
    • windowedEncoder

      public <T> org.apache.spark.sql.Encoder<WindowedValue<T>> windowedEncoder(String pCollectionId, org.apache.beam.model.pipeline.v1.RunnerApi.Components components)
      Encoder of the windowed values of a PCollection, derived from its wire coder.
    • computeOutputs

      public void computeOutputs()
      Evaluates every Dataset no transform consumed.
      Overrides:
      computeOutputs in class SparkTranslationContext