org.apache.beam.sdk.io.fileschematransform

Class FileReadSchemaTransformConfiguration

java.lang.Object

org.apache.beam.sdk.io.fileschematransform.FileReadSchemaTransformConfiguration

@DefaultSchema(value=AutoValueSchema.class)
public abstract class FileReadSchemaTransformConfiguration
extends java.lang.Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileReadSchemaTransformConfiguration.Builder

Field Summary

Fields

Modifier and Type	Field and Description
static java.util.Set<java.lang.String>	VALID_PROVIDERS

Constructor Summary

Constructors

Constructor and Description
FileReadSchemaTransformConfiguration()

Method Summary

Modifier and Type	Method and Description
static FileReadSchemaTransformConfiguration.Builder	builder()
abstract java.lang.String	getFilepattern() The filepattern used to match and read files.
abstract java.lang.String	getFormat() The format of the file(s) to read.
abstract java.lang.Long	getPollIntervalMillis() The time, in milliseconds, to wait before polling for new files.
java.lang.String	getSafeFilepattern()
java.lang.String	getSafeSchema()
abstract java.lang.String	getSchema() The schema used by sources to deserialize data and create Beam Rows.
abstract java.lang.Long	getTerminateAfterSecondsSinceNewOutput() If no new files are found after this many seconds, this transform will cease to watch for new files.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

VALID_PROVIDERS

public static final java.util.Set<java.lang.String> VALID_PROVIDERS

Constructor Detail

FileReadSchemaTransformConfiguration

public FileReadSchemaTransformConfiguration()

Method Detail

builder

public static FileReadSchemaTransformConfiguration.Builder builder()

getFormat

@SchemaFieldDescription(value="The format of the file(s) to read. Possible values are \"lines\", \"avro\", \"parquet\", \"json\".")
public abstract java.lang.String getFormat()

The format of the file(s) to read.

Possible values are: `"lines"`, `"avro"`, `"parquet"`, `"json"`

getFilepattern

@SchemaFieldDescription(value="The filepattern used to match and read files. May instead use an input PCollection<Row> of filepatterns. To do so, each Row must have a \"filepattern\" String field containing the filepattern.")
 @Nullable
public abstract java.lang.String getFilepattern()

The filepattern used to match and read files.

May instead use an input PCollection of filepatterns. To do so, each Row must have a "filepattern" String field containing the filepattern.

getSafeFilepattern

public java.lang.String getSafeFilepattern()

getSchema

@SchemaFieldDescription(value="The schema used by sources to deserialize data and create Beam Rows. May provide either a String representation of the schema or a single path to a file that contains the schema.")
 @Nullable
public abstract java.lang.String getSchema()

The schema used by sources to deserialize data and create Beam Rows.

May provide either a String representation of the schema or a single path to a file that contains the schema.

getSafeSchema

public java.lang.String getSafeSchema()

getPollIntervalMillis

@SchemaFieldDescription(value="The time, in milliseconds, to wait before polling for new files. This will set the pipeline to be a streaming pipeline that continuously watches for new files.Note: This only polls for new files. New updates to an existing file will not be watched for.")
 @Nullable
public abstract java.lang.Long getPollIntervalMillis()

The time, in milliseconds, to wait before polling for new files.

This will set the pipeline to be a streaming pipeline that continuously watches for new files.

Note: This only polls for new files. New updates to an existing file will not be watched for.

getTerminateAfterSecondsSinceNewOutput

@SchemaFieldDescription(value="If no new files are found after this many seconds, this transform will cease to watch for new files. The default is to never terminate. To set this parameter, a poll interval must also be provided.")
 @Nullable
public abstract java.lang.Long getTerminateAfterSecondsSinceNewOutput()

If no new files are found after this many seconds, this transform will cease to watch for new files.

The default is to never terminate. To set this parameter, a poll interval must also be provided.