Package org.apache.beam.sdk.io.delta

Class BeamParquetHandler

java.lang.Object

org.apache.beam.sdk.io.delta.BeamParquetHandler

All Implemented Interfaces:

ParquetHandler

public class BeamParquetHandler
extends Object
implements ParquetHandler

A Beam specific ParquetHandler that delegates row group claiming to a DeltaReadTaskTracker.

Constructor Summary

Constructors

Constructor	Description
BeamParquetHandler(org.apache.hadoop.conf.Configuration conf, ParquetHandler delegate, RestrictionTracker<OffsetRange,Long> tracker)

Method Summary

Modifier and Type	Method	Description
boolean	hasClaimFailed()	A method that is expected to be called after the first file processing is done.
CloseableIterator<FileReadResult>	readParquetFiles(CloseableIterator<FileStatus> fileIter, StructType physicalSchema, Optional<Predicate> predicate)
CloseableIterator<FileReadResult>	readParquetFiles(CloseableIterator<FileStatus> fileIter, StructType physicalSchema, Optional<Predicate> predicate, long startRgIndex)	Reads Parquet files starting from a given row group index.
void	writeParquetFileAtomically(String filePath, CloseableIterator<FilteredColumnarBatch> data)
CloseableIterator<DataFileStatus>	writeParquetFiles(String filePath, CloseableIterator<FilteredColumnarBatch> data, List<Column> statsColumns)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

BeamParquetHandler

public BeamParquetHandler(org.apache.hadoop.conf.Configuration conf,
 ParquetHandler delegate,
 RestrictionTracker<OffsetRange,Long> tracker)

Method Details

hasClaimFailed

public boolean hasClaimFailed()

A method that is expected to be called after the first file processing is done. It returns whether the last file process resulted in a claim failure. This allows the caller to skip trying to read the remaining files of the task which would result in claim failures for each row group within them.

Returns:

true, if the last file process resulted in a claim failure. Returns false otherwise.

readParquetFiles

public CloseableIterator<FileReadResult> readParquetFiles(CloseableIterator<FileStatus> fileIter,
 StructType physicalSchema,
 Optional<Predicate> predicate)
                                                   throws IOException

Specified by:

readParquetFiles in interface ParquetHandler

Throws:

IOException

readParquetFiles

public CloseableIterator<FileReadResult> readParquetFiles(CloseableIterator<FileStatus> fileIter,
 StructType physicalSchema,
 Optional<Predicate> predicate,
 long startRgIndex)
                                                   throws IOException

Reads Parquet files starting from a given row group index.

This takes the RestrictionTracker referenced by the current ParquetReader into consideration when reading by performing the following.

* Skips blocks of the set of files till the given start row group index or the start point of the RestrictionTracker, whatever is higher. * Invokes tryClaim when reading a specific block stops reading if a tryClaim fails. * Stops reading if the end of the range of the RestrictionTracker is reached.

If tryClaim fails during reading, subsequent hasClaimFailed calls will return true, so the caller can skip reading subsequent files that are in the range being considered for reading.

Throws:

IOException

writeParquetFileAtomically

public void writeParquetFileAtomically(String filePath,
 CloseableIterator<FilteredColumnarBatch> data)
                                throws IOException

Specified by:

writeParquetFileAtomically in interface ParquetHandler

Throws:

IOException

writeParquetFiles

public CloseableIterator<DataFileStatus> writeParquetFiles(String filePath,
 CloseableIterator<FilteredColumnarBatch> data,
 List<Column> statsColumns)
                                                    throws IOException

Specified by:

writeParquetFiles in interface ParquetHandler

Throws:

IOException