org.apache.beam.runners.spark.structuredstreaming.translation.batch

Class DatasetSourceBatch

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.translation.batch.DatasetSourceBatch

All Implemented Interfaces:

org.apache.spark.sql.connector.catalog.TableProvider

public class DatasetSourceBatch
extends java.lang.Object
implements org.apache.spark.sql.connector.catalog.TableProvider

Spark DataSourceV2 API was removed in Spark3. This is a Beam source wrapper using the new spark 3 source API.

Constructor Summary

Constructors

Constructor and Description
DatasetSourceBatch()

Method Summary

Modifier and Type	Method and Description
org.apache.spark.sql.connector.catalog.Table	getTable(org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitioning, java.util.Map<java.lang.String,java.lang.String> properties)
org.apache.spark.sql.types.StructType	inferSchema(org.apache.spark.sql.util.CaseInsensitiveStringMap options)
boolean	supportsExternalMetadata()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.sql.connector.catalog.TableProvider

inferPartitioning

Constructor Detail

DatasetSourceBatch

public DatasetSourceBatch()

Method Detail

inferSchema

public org.apache.spark.sql.types.StructType inferSchema(org.apache.spark.sql.util.CaseInsensitiveStringMap options)

Specified by:

inferSchema in interface org.apache.spark.sql.connector.catalog.TableProvider

supportsExternalMetadata

public boolean supportsExternalMetadata()

Specified by:

supportsExternalMetadata in interface org.apache.spark.sql.connector.catalog.TableProvider

getTable

public org.apache.spark.sql.connector.catalog.Table getTable(org.apache.spark.sql.types.StructType schema,
                                                             org.apache.spark.sql.connector.expressions.Transform[] partitioning,
                                                             java.util.Map<java.lang.String,java.lang.String> properties)

Specified by:

getTable in interface org.apache.spark.sql.connector.catalog.TableProvider