org.apache.beam.sdk.extensions.protobuf

Class ProtoSchemaTranslator

java.lang.Object

org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator

@Experimental(value=SCHEMAS)
public class ProtoSchemaTranslator
extends java.lang.Object

This class provides utilities for inferring a Beam schema from a protocol buffer.

The following proto primitive types map to the following Beam types:

• INT32 maps to FieldType.INT32
• INT64 maps to FieldType.INT64
• FLOAT maps to FieldType.FLOAT
• DOUBLE maps to FieldType.DOUBLE
• BOOL maps to FieldType.BOOLEAN
• STRING maps to FieldType.STRING
• BYTES maps to FieldType.BYTES

The following proto numeric types do not have have native Beam primitive types. LogicalType objects were created to represent these types. Normal numeric types are used as the base type of each of these logical types, so SQL queries should work as normal.

• UINT32 maps to FieldType.logicalType(new ProtoSchemaLogicalTypes.UInt32()))
• SINT32 maps to FieldType.logicalType(new ProtoSchemaLogicalTypes.SInt32()))
• FIXED32 maps to FieldType.logicalType(new ProtoSchemaLogicalTypes.Fixed32()))
• SFIXED32 maps to FieldType.logicalType(new ProtoSchemaLogicalTypes.SFixed32()))
• UINT64 maps to FieldType.logicalType(new ProtoSchemaLogicalTypes.UInt64()))
• SINT64 maps to FieldType.logicalType(new ProtoSchemaLogicalTypes.SInt64()))
• FIXED64 maps to FieldType.logicalType(new ProtoSchemaLogicalTypes.Fixed64()))
• SFIXED64 maps to FieldType.logicalType(new ProtoSchemaLogicalTypes.SFixed64()))

Protobuf maps are mapped to Beam FieldType.MAP types. Protobuf repeated fields are mapped to Beam FieldType.ARRAY types.

Beam schemas include the EnumerationType logical type to represent enumerations, and protobuf enumerations are translated to this logical type. The base representation type for this logical type is an INT32.

Beam schemas include the OneOfType logical type to represent unions, and protobuf oneOfs are translated to this logical type. The base representation type for this logical type is a subrow containing an optional field for each oneof option.

google.com.protobuf.Timestamp messages cannot be translated to FieldType.DATETIME types, as the proto type represents nanonseconds and Beam's native type does not currently support that. a new TimestampNanos logical type has been introduced to allow representing nanosecond timestamp, as well as a DurationNanos logical type to represent google.com.protobuf.Duration types.

Protobuf wrapper classes are translated to nullable types, as follows.

• google.protobuf.Int32Value maps to a nullable FieldType.INT32
• google.protobuf.Int64Value maps to a nullable FieldType.INT64
• google.protobuf.UInt32Value maps to a nullable FieldType.logicalType(new UInt32())
• google.protobuf.UInt64Value maps to a nullable Field.logicalType(new UInt64())
• google.protobuf.FloatValue maps to a nullable FieldType.FLOAT
• google.protobuf.DoubleValue maps to a nullable FieldType.DOUBLE
• google.protobuf.BoolValue maps to a nullable FieldType.BOOLEAN
• google.protobuf.StringValue maps to a nullable FieldType.STRING
• google.protobuf.BytesValue maps to a nullable FieldType.BYTES

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	PROTO_NUMBER_METADATA_TAG This METADATA tag is used to store the field number of a proto tag.

Constructor Summary

Constructors

Constructor and Description
ProtoSchemaTranslator()

Method Summary

Modifier and Type	Method and Description
static int	getFieldNumber(Schema.FieldType fieldType) Return the proto field number for a type.
static Schema	getSchema(java.lang.Class<? extends Message> clazz) Return a Beam scheam representing a proto class.
static Schema.FieldType	withFieldNumber(Schema.FieldType fieldType, int index) Attach a proto field number to a type.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PROTO_NUMBER_METADATA_TAG

public static final java.lang.String PROTO_NUMBER_METADATA_TAG

This METADATA tag is used to store the field number of a proto tag.

See Also:

Constant Field Values

Constructor Detail

ProtoSchemaTranslator

public ProtoSchemaTranslator()

Method Detail

withFieldNumber

public static Schema.FieldType withFieldNumber(Schema.FieldType fieldType,
                                               int index)

Attach a proto field number to a type.

getFieldNumber

public static int getFieldNumber(Schema.FieldType fieldType)

Return the proto field number for a type.

getSchema

public static Schema getSchema(java.lang.Class<? extends Message> clazz)

Return a Beam scheam representing a proto class.