Package org.apache.beam.sdk.io.cdap

Class Plugin<K,V>

java.lang.Object
org.apache.beam.sdk.io.cdap.Plugin<K,V>
public abstract class Plugin<K,V> extends Object
Class wrapper for a CDAP plugin.

  • Field Details

  • Constructor Details

    • Plugin

      public Plugin()

  • Method Details

    • getContext

      public abstract BatchContextImpl getContext()
      Gets the context of a plugin.

    • getPluginClass

      public abstract Class<?> getPluginClass()
      Gets the main class of a plugin.

    • getFormatClass

      @Nullable public abstract Class<?> getFormatClass()
      Gets InputFormat or OutputFormat class for a plugin.

    • getFormatProviderClass

      @Nullable public abstract Class<?> getFormatProviderClass()
      Gets InputFormatProvider or OutputFormatProvider class for a plugin.

    • getReceiverClass

      @Nullable public abstract Class<? extends org.apache.spark.streaming.receiver.Receiver<V>> getReceiverClass()
      Gets Spark Receiver class for a CDAP plugin.

    • getGetOffsetFn

      @Nullable public abstract SerializableFunction<V,Long> getGetOffsetFn()
      Gets a SerializableFunction that defines how to get record offset for CDAP Plugin class.

    • getGetReceiverArgsFromConfigFn

      @Nullable public abstract SerializableFunction<PluginConfig,Object[]> getGetReceiverArgsFromConfigFn()
      Gets a SerializableFunction that defines how to get constructor arguments for Receiver using PluginConfig.

    • withConfig

      public Plugin<K,V> withConfig(PluginConfig pluginConfig)
      Sets a plugin config.

    • getPluginConfig

      @Nullable public PluginConfig getPluginConfig()
      Gets a plugin config.

    • prepareRun

      public void prepareRun()
      Calls SubmitterLifecycle.prepareRun(Object) method on the cdapPluginObj passing needed configuration object as a parameter. This method is needed for validating connection to the CDAP sink/source and performing initial tuning.

    • withHadoopConfiguration

      public Plugin<K,V> withHadoopConfiguration(Class<K> formatKeyClass, Class<V> formatValueClass)
      Sets a plugin Hadoop configuration.

    • withHadoopConfiguration

      public Plugin<K,V> withHadoopConfiguration(org.apache.hadoop.conf.Configuration hadoopConfiguration)
      Sets a plugin Hadoop configuration.

    • getHadoopConfiguration

      public org.apache.hadoop.conf.Configuration getHadoopConfiguration()
      Gets a plugin Hadoop configuration.

    • getPluginType

      public abstract PluginConstants.PluginType getPluginType()
      Gets a plugin type.

    • initPluginType

      public static PluginConstants.PluginType initPluginType(Class<?> pluginClass) throws IllegalArgumentException
      Gets value of a plugin type.
      Throws:
      IllegalArgumentException

    • initContext

      public static BatchContextImpl initContext(Class<?> cdapPluginClass)
      Initializes BatchContextImpl for CDAP plugin.

    • isUnbounded

      public Boolean isUnbounded()
      Gets value of a plugin type.

    • getReceiverBuilder

      public ReceiverBuilder<V,? extends org.apache.spark.streaming.receiver.Receiver<V>> getReceiverBuilder()
      Gets a ReceiverBuilder.

    • createBatch

      public static <K, V> Plugin<K,V> createBatch(Class<?> newPluginClass, Class<?> newFormatClass, Class<?> newFormatProviderClass)
      Creates a batch plugin instance.
      Parameters:
      newPluginClass - class of the CDAP plugin Plugin.
      newFormatClass - Hadoop Input or Output format class.
      newFormatProviderClass - Hadoop Input or Output format provider class.

    • createStreaming

      public static <K, V> Plugin<K,V> createStreaming(Class<?> newPluginClass, SerializableFunction<V,Long> getOffsetFn, Class<? extends org.apache.spark.streaming.receiver.Receiver<V>> receiverClass, SerializableFunction<PluginConfig,Object[]> getReceiverArgsFromConfigFn)
      Creates a streaming plugin instance.
      Parameters:
      newPluginClass - class of the CDAP plugin Plugin.
      getOffsetFn - SerializableFunction that defines how to get record offset for CDAP Plugin class.
      receiverClass - Spark Receiver class for a CDAP plugin.
      getReceiverArgsFromConfigFn - SerializableFunction that defines how to get constructor arguments for Receiver using PluginConfig.

    • createStreaming

      public static <K, V> Plugin<K,V> createStreaming(Class<?> newPluginClass, SerializableFunction<V,Long> getOffsetFn, Class<? extends org.apache.spark.streaming.receiver.Receiver<V>> receiverClass)
      Creates a streaming plugin instance with default function for getting args for Receiver.
      Parameters:
      newPluginClass - class of the CDAP plugin Plugin.
      getOffsetFn - SerializableFunction that defines how to get record offset for CDAP Plugin class.
      receiverClass - Spark Receiver class for a CDAP plugin.

    • builder

      public static <K, V> Plugin.Builder<K,V> builder()
      Creates a plugin builder instance.