org.apache.beam.sdk.io

Class TextSource

java.lang.Object

org.apache.beam.sdk.io.Source<T>

org.apache.beam.sdk.io.BoundedSource<T>

org.apache.beam.sdk.io.OffsetBasedSource<T>

org.apache.beam.sdk.io.FileBasedSource<java.lang.String>

org.apache.beam.sdk.io.TextSource

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

public class TextSource
extends FileBasedSource<java.lang.String>

Implementation detail of TextIO.Read.

A FileBasedSource which can decode records delimited by newline characters.

This source splits the data into records using UTF-8 \n, \r, or \r\n as the delimiter. This source is not strict and supports decoding the last record even if it is not delimited. Finally, no records are decoded if the stream is empty.

This source supports reading from any arbitrary byte position within the stream. If the starting position is not 0, then bytes are skipped until the first delimiter is found representing the beginning of the first record to be decoded.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.FileBasedSource

FileBasedSource.FileBasedReader<T>, FileBasedSource.Mode

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.OffsetBasedSource

OffsetBasedSource.OffsetBasedReader<T>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.BoundedSource

BoundedSource.BoundedReader<T>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.Source

Source.Reader<T>

Constructor Summary

Constructors

Constructor and Description
TextSource(MatchResult.Metadata metadata, long start, long end, byte[] delimiter)
TextSource(MatchResult.Metadata metadata, long start, long end, byte[] delimiter, int skipHeaderLines)
TextSource(ValueProvider<java.lang.String> fileSpec, EmptyMatchTreatment emptyMatchTreatment, byte[] delimiter)
TextSource(ValueProvider<java.lang.String> fileSpec, EmptyMatchTreatment emptyMatchTreatment, byte[] delimiter, int skipHeaderLines)

Method Summary

Modifier and Type	Method and Description
protected FileBasedSource<java.lang.String>	createForSubrangeOfFile(MatchResult.Metadata metadata, long start, long end) Creates and returns a new FileBasedSource of the same type as the current FileBasedSource backed by a given file and an offset range.
protected FileBasedSource.FileBasedReader<java.lang.String>	createSingleFileReader(PipelineOptions options) Creates and returns an instance of a FileBasedReader implementation for the current source assuming the source represents a single file.
Coder<java.lang.String>	getOutputCoder() Returns the Coder to use for the data read from this source.

Methods inherited from class org.apache.beam.sdk.io.FileBasedSource

createReader, createSourceForSubrange, getEmptyMatchTreatment, getEstimatedSizeBytes, getFileOrPatternSpec, getFileOrPatternSpecProvider, getMaxEndOffset, getMode, getSingleFileMetadata, isSplittable, populateDisplayData, split, toString, validate

Methods inherited from class org.apache.beam.sdk.io.OffsetBasedSource

getBytesPerOffset, getEndOffset, getMinBundleSize, getStartOffset

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

TextSource

public TextSource(ValueProvider<java.lang.String> fileSpec,
                  EmptyMatchTreatment emptyMatchTreatment,
                  byte[] delimiter,
                  int skipHeaderLines)

TextSource

public TextSource(ValueProvider<java.lang.String> fileSpec,
                  EmptyMatchTreatment emptyMatchTreatment,
                  byte[] delimiter)

TextSource

public TextSource(MatchResult.Metadata metadata,
                  long start,
                  long end,
                  byte[] delimiter,
                  int skipHeaderLines)

TextSource

public TextSource(MatchResult.Metadata metadata,
                  long start,
                  long end,
                  byte[] delimiter)

Method Detail

createForSubrangeOfFile

protected FileBasedSource<java.lang.String> createForSubrangeOfFile(MatchResult.Metadata metadata,
                                                                    long start,
                                                                    long end)

Description copied from class: FileBasedSource

Creates and returns a new FileBasedSource of the same type as the current FileBasedSource backed by a given file and an offset range. When current source is being split, this method is used to generate new sub-sources. When creating the source subclasses must call the constructor #FileBasedSource(Metadata, long, long, long) of FileBasedSource with corresponding parameter values passed here.

Specified by:

createForSubrangeOfFile in class FileBasedSource<java.lang.String>

Parameters:

metadata - file backing the new FileBasedSource.

start - starting byte offset of the new FileBasedSource.

end - ending byte offset of the new FileBasedSource. May be Long.MAX_VALUE, in which case it will be inferred using FileBasedSource.getMaxEndOffset(org.apache.beam.sdk.options.PipelineOptions).

createSingleFileReader

protected FileBasedSource.FileBasedReader<java.lang.String> createSingleFileReader(PipelineOptions options)

Description copied from class: FileBasedSource

Creates and returns an instance of a FileBasedReader implementation for the current source assuming the source represents a single file. File patterns will be handled by FileBasedSource implementation automatically.

Specified by:

createSingleFileReader in class FileBasedSource<java.lang.String>

getOutputCoder

public Coder<java.lang.String> getOutputCoder()

Description copied from class: Source

Returns the Coder to use for the data read from this source.

Overrides:

getOutputCoder in class Source<java.lang.String>