Package org.apache.beam.runners.dataflow.internal

Class IsmFormat.IsmRecordCoder<V>

java.lang.Object
org.apache.beam.sdk.coders.Coder<IsmFormat.IsmRecord<V>>
org.apache.beam.sdk.coders.CustomCoder<IsmFormat.IsmRecord<V>>
org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecordCoder<V>
All Implemented Interfaces:
Serializable
Enclosing class:
IsmFormat
public static class IsmFormat.IsmRecordCoder<V> extends CustomCoder<IsmFormat.IsmRecord<V>>
A Coder for IsmFormat.IsmRecords.

Note that this coder standalone will not produce an Ism file. This coder can be used to materialize a PCollection of IsmFormat.IsmRecords. Only when this coder is combined with an IsmSink will one produce an Ism file.

The IsmFormat.IsmRecord encoded format is:

  • encoded key component 1 using key component coder 1
  • ...
  • encoded key component N using key component coder N
  • encoded value using value coder
See Also:

  • Method Details

    • of

      public static <V> IsmFormat.IsmRecordCoder<V> of(int numberOfShardKeyCoders, int numberOfMetadataShardKeyCoders, List<Coder<?>> keyComponentCoders, Coder<V> valueCoder)
      Returns an IsmRecordCoder with the specified key component coders, value coder.

    • getKeyComponentCoders

      public List<Coder<?>> getKeyComponentCoders()
      Returns the list of key component coders.

    • getKeyComponentCoder

      public <T> Coder<T> getKeyComponentCoder(int index)
      Returns the key coder at the specified index.

    • getValueCoder

      public Coder<V> getValueCoder()
      Returns the value coder.

    • encode

      public void encode(IsmFormat.IsmRecord<V> value, OutputStream outStream) throws CoderException, IOException
      Description copied from class: Coder
      Encodes the given value of type T onto the given output stream. Multiple elements can be encoded next to each other on the output stream, each coder should encode information to know how many bytes to read when decoding. A common approach is to prefix the encoding with the element's encoded length.
      Specified by:
      encode in class Coder<IsmFormat.IsmRecord<V>>
      Throws:
      CoderException - if the value could not be encoded for some reason
      IOException - if writing to the OutputStream fails for some reason

    • decode

      public IsmFormat.IsmRecord<V> decode(InputStream inStream) throws CoderException, IOException
      Description copied from class: Coder
      Decodes a value of type T from the given input stream in the given context. Returns the decoded value. Multiple elements can be encoded next to each other on the input stream, each coder should encode information to know how many bytes to read when decoding. A common approach is to prefix the encoding with the element's encoded length.
      Specified by:
      decode in class Coder<IsmFormat.IsmRecord<V>>
      Throws:
      CoderException - if the value could not be decoded for some reason
      IOException - if reading from the InputStream fails for some reason

    • getNumberOfShardKeyCoders

      public int getNumberOfShardKeyCoders(List<?> keyComponents)

    • hash

      public int hash(List<?> keyComponents)
      Computes the shard id for the given key component(s).

      The shard keys are encoded into their byte representations and hashed using the 32-bit murmur3 algorithm, x86 variant (little-endian variant), using 1225801234 as the seed value. We ensure that shard ids for metadata keys and normal keys do not overlap.

    • encodeAndHash

      public int encodeAndHash(List<?> keyComponents, RandomAccessData keyBytesToMutate)
      Computes the shard id for the given key component(s).

      Mutates keyBytes such that when returned, contains the encoded version of the key components.

    • encodeAndHash

      public int encodeAndHash(List<?> keyComponents, RandomAccessData keyBytesToMutate, List<Integer> keyComponentByteOffsetsToMutate)
      Computes the shard id for the given key component(s).

      Mutates keyBytes such that when returned, contains the encoded version of the key components. Also, mutates keyComponentByteOffsetsToMutate to store the location where each key component's encoded byte representation ends within keyBytes.

    • getCoderArguments

      public List<Coder<?>> getCoderArguments()
      Description copied from class: CustomCoder
      If this is a Coder for a parameterized type, returns the list of Coders being used for each of the parameters in the same order they appear within the parameterized type's type signature. If this cannot be done, or this Coder does not encode/decode a parameterized type, returns the empty list..

      Returns an empty list. A CustomCoder has no default argument coders.

      Overrides:
      getCoderArguments in class CustomCoder<IsmFormat.IsmRecord<V>>

    • verifyDeterministic

      public void verifyDeterministic() throws Coder.NonDeterministicException
      Description copied from class: CustomCoder
      Throw Coder.NonDeterministicException if the coding is not deterministic.

      In order for a Coder to be considered deterministic, the following must be true:

      • two values that compare as equal (via Object.equals() or Comparable.compareTo(), if supported) have the same encoding.
      • the Coder always produces a canonical encoding, which is the same for an instance of an object even if produced on different computers at different times.
      Overrides:
      verifyDeterministic in class CustomCoder<IsmFormat.IsmRecord<V>>
      Throws:
      Coder.NonDeterministicException - a CustomCoder is presumed nondeterministic.

    • consistentWithEquals

      public boolean consistentWithEquals()
      Description copied from class: Coder
      Returns true if this Coder is injective with respect to Object.equals(java.lang.Object).

      Whenever the encoded bytes of two values are equal, then the original values are equal according to Objects.equals(). Note that this is well-defined for null.

      This condition is most notably false for arrays. More generally, this condition is false whenever equals() compares object identity, rather than performing a semantic/structural comparison.

      By default, returns false.

      Overrides:
      consistentWithEquals in class Coder<IsmFormat.IsmRecord<V>>

    • structuralValue

      public Object structuralValue(IsmFormat.IsmRecord<V> record)
      Description copied from class: Coder
      Returns an object with an Object.equals() method that represents structural equality on the argument.

      For any two values x and y of type T, if their encoded bytes are the same, then it must be the case that structuralValue(x).equals(structuralValue(y)).

      Most notably:

      • The structural value for an array coder should perform a structural comparison of the contents of the arrays, rather than the default behavior of comparing according to object identity.
      • The structural value for a coder accepting null should be a proper object with an equals() method, even if the input value is null.

      See also Coder.consistentWithEquals().

      By default, if this coder is Coder.consistentWithEquals(), and the value is not null, returns the provided object. Otherwise, encodes the value into a byte[], and returns an object that performs array equality on the encoded bytes.

      Overrides:
      structuralValue in class Coder<IsmFormat.IsmRecord<V>>

    • equals

      public boolean equals(@Nullable Object other)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object