Package org.apache.beam.sdk.io.iceberg

Class SerializableDataFile

java.lang.Object

org.apache.beam.sdk.io.iceberg.SerializableDataFile

@DefaultSchema(AutoValueSchema.class)
public abstract class SerializableDataFile
extends Object

Serializable version of an Iceberg DataFile.

DataFile is not serializable and the Iceberg API doesn't offer an easy way to encode/decode it. This class is an identical version that can be used as a PCollection element type.

NOTE: If you add any new fields here, you need to also update the equals(java.lang.Object) and hashCode() methods.

Use from(DataFile, String) to create a SerializableDataFile and createDataFile(Map) to reconstruct the original DataFile.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	SerializableDataFile.Builder

Constructor Summary

Constructors

Constructor	Description
SerializableDataFile()

Method Summary

Modifier and Type	Method	Description
static SerializableDataFile.Builder	builder()
org.apache.iceberg.DataFile	createDataFile(Map<Integer,org.apache.iceberg.PartitionSpec> partitionSpecs)	Reconstructs the original DataFile from this SerializableDataFile.
final boolean	equals(@Nullable Object o)
static SerializableDataFile	from(org.apache.iceberg.DataFile f, String partitionPath)
static SerializableDataFile	from(org.apache.iceberg.DataFile f, String partitionPath, boolean includeMetrics)	Create a SerializableDataFile from a DataFile and its associated PartitionKey.
abstract @Nullable Long	getDataSequenceNumber()
abstract @Nullable Long	getFileSequenceNumber()
abstract long	getFileSizeInBytes()
abstract @Nullable Long	getFirstRowId()
abstract @Nullable Map<Integer,byte[]>	getLowerBounds()
abstract String	getPartitionPath()
abstract String	getPath()
abstract @Nullable Map<Integer,byte[]>	getUpperBounds()
final int	hashCode()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SerializableDataFile

public SerializableDataFile()

Method Details

builder

public static SerializableDataFile.Builder builder()

getPath

@SchemaFieldNumber("0")
public abstract String getPath()

getFileSizeInBytes

@SchemaFieldNumber("3")
public abstract long getFileSizeInBytes()

getPartitionPath

@SchemaFieldNumber("4")
public abstract String getPartitionPath()

getLowerBounds

@SchemaFieldNumber("12")
public abstract @Nullable Map<Integer,byte[]> getLowerBounds()

getUpperBounds

@SchemaFieldNumber("13")
public abstract @Nullable Map<Integer,byte[]> getUpperBounds()

getDataSequenceNumber

@SchemaFieldNumber("14")
public abstract @Nullable Long getDataSequenceNumber()

getFileSequenceNumber

@SchemaFieldNumber("15")
public abstract @Nullable Long getFileSequenceNumber()

getFirstRowId

@SchemaFieldNumber("16")
public abstract @Nullable Long getFirstRowId()

from

public static SerializableDataFile from(org.apache.iceberg.DataFile f,
 String partitionPath)

from

public static SerializableDataFile from(org.apache.iceberg.DataFile f,
 String partitionPath,
 boolean includeMetrics)

Create a SerializableDataFile from a DataFile and its associated PartitionKey.

createDataFile

public org.apache.iceberg.DataFile createDataFile(Map<Integer,org.apache.iceberg.PartitionSpec> partitionSpecs)

Reconstructs the original DataFile from this SerializableDataFile.

We require an input PartitionSpec as well because there's no easy way to reconstruct it from Beam-compatible types.

equals

public final boolean equals(@Nullable Object o)

Overrides:

equals in class Object

hashCode

public final int hashCode()

Overrides:

hashCode in class Object